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Abstract 


The  use  of  Extremely  Agile  Micro  Aerial  Vehicles  (EA-MAVs)  drives  the  need  for  a  small  and 
light  controller  which  will  not  hinder  a  soldier  carrying  it.  This  requirement  brings  an  issue  of 
designing  an  effective  operator  interface  coupled  with  the  controller.  Little  human  factors  work 
has  been  done  on  what  the  most  effective  method  is  for  controlling  EA-MAVs  using  a  handheld 
device.  To  investigate  design  methods  for  the  development  of  an  interface  which  must  be  intuitive 
in  function  and  easy  to  learn  for  an  average  soldier,  DRDC  Toronto  conducted  an  experiment  to 
evaluate  interface  display  mode  and  command  control  input  method.  Display  mode  compared  a 
display  that  showed  both  a  sensor  view  and  a  map  view  (simultaneous)  with  a  display  that 
showed  one  view  at  a  time  (sequential).  Command  control  input  method  compared  two  types  of 
input  control  methods:  touch  screen  and  tactile  buttons.  Forty  four  (44)  subjects  participated  in 
the  experiment  and  navigated  a  virtual  EA-MAV  through  specified  waypoints  in  an  urban  area 
and  a  building.  Subjects’  performance  was  measured  against  six  dependent  variables:  (1)  situation 
awareness,  (2)  display  switch  frequency  (between  sensor  view  and  map  view),  (3)  task 
completion  time,  (4)  mental  workload,  (5)  trajectory  error  of  the  flight,  and  (6)  training  time.  The 
results  revealed  that  the  simultaneous  display  and  the  touch  screen  control  are  the  optimal  design 
methods  for  the  handheld  interface  to  be  used  easily  when  maintaining  situation  awareness.  The 
findings  provided  guidance  for  designing  operator  interfaces  on  handheld  devices  and  further 
facilitated  the  development  of  a  statement  of  requirements  of  EA-MAV  systems. 
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Resume 


L’utilisation  des  Extremement  agile  Micro  Aerial  Vehicles  (EA-MAV)  entraine  la  necessite  d’un 
controleur  petit  et  leger  a  apporter  qui  ne  reduira  pas  la  performance  du  soldat.  Cette  condition  a 
amene  a  concevoir  un  interface  operateur  couple  avec  le  controleur.  Une  recherche  de  facteurs 
humains  qui  etait  effectue  sur  la  methode  la  plus  efficace,  est  developpe  pour  controler  EA- 
MAVs  en  utilisant  un  dispositif  portable.  Pour  investiguer  le  design  des  methodes  en 
developpement  d’un  interface  qui  doit  etre  intuitif  dans  la  fonction  et  facile  a  apprendre  pour  un 
soldat  moyen,  RDDC  Toronto  a  entrepris  une  experience  pour  evaluer  l'efficacite  du  mode 
d’affichage  et  la  methode  du  commande  de  controle  d'entree.  Le  mode  d’affichage  a  compare  un 
affichage  qui  combine  une  vue  due  capteur  et  une  vue  de  carte  (simultanee),  avec  un  affichage  qui 
separe  les  deux  points  de  vue  (sequentiel).  La  methode  du  commande  de  controle  d'entree  a 
compare  deux  types  de  methodes  de  controle  d'entree:  ecran  tactile  et  des  boutons  tactiles. 
Quarante-quatre  (44)  sujets  ont  participe  a  l'experience  et  ont  navigue  une  EA-MAV  virtuelle  en 
passant  par  des  points-route  specifiques,  dans  une  zone  urbaine  et  un  batiment.  La  performance 
des  sujets  etait  mesure  par  rapport  a  six  variables  dependantes.  (1)  conscience  de  la  situation  (2) 
affichage  de  la  frequence  de  commutation  (entre  la  vue  de  capteur  et  de  carte),  (3)  le  temps 
l'execution  des  taches,  (4)  charge  de  travail  mental,  (5)  erreur  de  trajectoire  du  vol,  et  (6)  le  temps 
de  formation.  L'experience  a  revele  que  l'affichage  simultane  et  l'ecran  tactile  de  controle  sont  les 
methodes  de  conception  les  plus  optimal  pour  soit  utiliser  facilement  que  Tinterface  portable  et 
facile  pour  un  utilisation  facile  en  maintenant  la  conscience  de  situation.  Ces  resultats  foumissent 
un  guide  pour  la  conception  d'interface  d’ operateur  dans  dispositifs  portables  et  ont  facilite  le 
developpement  des  conditions  des  systemes  d’  EA-MAV. 
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Executive  summary 


An  Empirical  Study  on  Operator  Interface  Design  for  Handheld 
Devices  to  Control  Micro  Aerial  Vehicles: 

Ming  Hou;  Sheila  Young;  Shi  Yin;  Joshua  Ru  Selvadurai  DRDC  Toronto  TR 
2010-075;  Defence  R&D  Canada  -  Toronto;  October  2010. 


Extremely  Agile  Micro  Aerial  Vehicles  (EA-MAVs)  are  miniature  flying  machines  that  will 
provide  “over-the-hill”  or  “around-the-next-house”  reconnaissance  to  smaller  combat  elements 
such  as  infantry  sections.  This  system  is  expected  to  be  used  by  the  Canadian  Forces  (CF)  to 
improve  local  situation  awareness  (SA)  in  an  urban  setting.  To  better  use  such  technology, 
soldiers  will  require  a  suitable  Ground  Control  System  (GCS)  to  interface  with  the  flying  device 
they  are  operating.  Although  a  number  of  handheld  devices  have  been  developed  to  serve  as  a 
GCS  for  the  control  of  EA-MAVs,  there  is  no  operator  interface  design  guidance  for  a  GCS  on  a 
handheld  device.  Consequently,  there  is  a  requirement  for  Human  Factors  research  to  be  carried 
out  to  investigate  an  optimal  interface  design  method  for  the  control  of  EA-MAVs  using  a 
handheld  device.  A  previous  focus  group  study  has  looked  into  general  system  operational 
requirements  and  desired  interface  features.  The  study  results  suggested  a  further  investigation  on 
interface  display  mode  and  command  control  input  method  as  they  are  directly  related  to  the 
control  of  the  EA-MAV  system  and  maintaining  of  SA.  Thus,  the  objective  of  this  research  is  to 
explore  an  optimal  display  mode  and  a  command  control  input  method. 

Forty  four  (44)  subjects  participated  in  an  experiment  using  a  handheld  GCS  to  navigate  a  virtual 
EA-MAV  through  specified  waypoints  in  an  urban  area  and  a  building.  Interface  display  mode 
and  command  control  input  method  were  evaluated  on  subjects’  performance.  Display  mode 
compared  a  display  that  had  a  sensor  view  and  a  map  view  side  by  side  (simultaneous)  with  a 
display  that  showed  one  view  once  at  a  time  (sequential).  Command  control  input  method 
compared  two  types  of  input  control  methods:  touch  screen  and  tactile  buttons.  Subjects’ 
performance  was  measured  against  six  dependent  variables:  (1)  situation  awareness,  (2)  display 
switch  frequency  (between  sensor  view  and  map  view),  (3)  task  completion  time,  (4)  mental 
workload,  (5)  trajectory  error  of  the  flight,  and  (6)  training  time. 

The  results  revealed  that  the  display  mode  and  control  type  affected  EA-MAV  operations.  The 
simultaneous  display  provided  better  situation  awareness  and  produced  lower  display  switch 
frequency  than  the  sequential  display.  Touch  screen  control  resulted  in  a  faster  task  completion 
time  than  the  tactile  button  control.  Participants  reported  lower  mental  workload  with  the  touch 
screen  control  than  with  the  tactile  button  control.  These  findings  can  be  used  as  development 
recommendations  for  a  statement  of  requirements  and  interface  design  guidance  for  general  EA- 
MAV  systems.  For  better  situation  awareness  the  interface  should  be  able  to  display  both  sensor 
view  and  map  view  simultaneously.  For  the  ease-of-control,  a  handheld  device  with  a  touch 
screen  is  preferable  over  a  device  with  tactile  buttons. 
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Sommaire 


Une  Etude  Empirique  sur  la  Conception  d’un  Interface  Operateur  dans 
les  dispositifs  portables  pour  controler  les  Micro-drones  (MAV): 

Ming  Hou;  Sheila  Young;  Shi  Yin;  Joshua  Ru  Selvadurai  DRDC  Toronto  TR 

2010-075;  R  &  D  pour  la  defense  Canada  -  Toronto;  Mai  2010. 

Pour  une  meilleure  utilisation  d’une  telle  technologie,  les  soldats  ont  besoin  d’un  systeme  de 
controle  au  sol  (GCS)  approprie  pour  communiquer  avec  le  dispositif  de  vol.  Bien  que  un 
nombre  des  dispositifs  portables  ont  ete  developpes  pour  servir  comme  un  GCS  pour  le  controle 
d’EA-MAVs,  il  n’y  a  aucune  conception  d'un  guide  d’interface  d’operateur  pour  un  GCS,  sur  un 
dispositif  portable.  Par  consequent,  il  y  a  une  exigence  d’une  recherche  des  Facteurs  Humains  a 
mettre  en  oeuvre  pour  realiser  une  interface  utilisateur  le  plus  efficace  pour  le  controle  de  EA- 
MAV  en  utilisant  un  dispositif  portable.  Une  etude  precedente  a  examine  les  conditions 
operationnelles  du  systeme  general  et  les  caracteristiques  des  dispositifs  desires.  D’apres  les 
resultats  d'etude,  une  recherche  plus  appro fondies  sur  1’ interface  du  mode  d’affichage  et  la 
commande  de  controle  d'entree  est  necessaire  comme  ils  sont  directement  lies  au  controle  du 
systeme  d'EA-MAV  et  au  maintenance  du  SA.  Ainsi,  l'objectif  de  cette  recherche  est  d'explorer 
une  methode  optimal  du  mode  d'affichage  et  commande  de  controle  d'entree. 

Quarante-quatre  (44)  sujets  ont  participe  a  une  experience  en  utilisant  un  GCS  portable  pour 
naviger  une  EA-MAV  virtuelle  en  passant  par  des  points-route  specifiques,  dans  une  zone  urbaine 
et  un  hatiment  L’interface  du  mode  d’affichage  et  la  methode  du  commande  de  controle  d’entree 
ont  ete  evalues  a  partir  de  la  performance  des  sujets.  Le  mode  d’affichage  a  compare  un  affichage 
qui  combine  une  vue  de  capteur  et  une  vue  de  carte  (simultanee),  avec  un  affichage  qui  separe  les 
deux  points  de  vue  (sequentiel).  La  methode  du  commande  de  controle  d’entree  a  compare  a 
compare  deux  types  de  methodes  de  controle  d'entree:  ecran  tactile  et  des  boutons  tactiles.  La 
performance  des  sujets  etait  mesure  par  rapport  a  six  variables  dependantes.  (1)  conscience  de  la 
situation  (2)  affichage  de  la  frequence  de  commutation  (entre  la  vue  de  capteur  et  de  carte),  (3)  le 
temps  l'execution  des  taches,  (4)  charge  de  travail  mental,  (5)  erreur  de  trajectoire  du  vol,  et  (6)  le 
temps  de  formation. 

L'experience  revele  que  le  mode  d'affichage  et  le  type  de  controle  ont  des  effets  sur  le 
fonctionnement  des  operations  d’un  EA-MAV.  Affichage  simultane  fournit  une  conscience  de  la 
situation  plus  eleves  et  une  frequence  commutateur  d'affichage  plus  bas,  par  rapport  a  l'affichage 
sequentiel.  Controle  d’ecran  tactile  montre  un  delai  d'execution  des  taches  plus  rapidement  que  le 
controle  de  bouton  tactile.  Les  participants  ont  signale  une  baisse  de  la  charge  mentale  avec 
l'ecran  tactile  qu'avec  le  controle  bouton  tactile.  Ces  resultats  foumissent  des  orientations  pour  la 
conception  d'interface  utilisateur  pour  les  systemes  generates  d’  EA-MAV.  Pour  avoir  une 
meilleure  conscience  de  situation,  l'interface  doit  etre  capable  d’afficher  une  vue  de  capteur  et  une 
vue  de  carte  simultanement.  Pour  le  facilite-de  controle,  un  dispositif  portable  avec  un  ecran 
tactile  est  plus  preferable  qu’un  dispositif  avec  des  boutons  tactiles. 
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1  Introduction 


Micro  Aerial  Vehicles  (MAVs)  are  miniature  flying  machines  that  will  provide  “over-the-hill”  or 
“around-the-next-house”  reconnaissance  to  smaller  combat  elements  such  as  infantry  sections. 
The  Extremely  Agile  Micro  Aerial  Vehicle  (EA-MAV)  currently  under  development  by  DRDC  is 
an  electric,  fixed-wing  aircraft  equipped  with  a  video  sensor  and  an  audio  sensor  (Ste-Croix  & 
Angel,  2008).  The  forward  flight  speed  of  this  MAV  varies  from  a  standstill  to  a  running  pace.  In 
the  hovering  mode  with  the  nose  pointed  vertically  up,  it  can  roll  about  its  longitudinal  axis  to 
scan  its  surroundings  with  the  video  sensor.  Using  onboard  electronics,  the  MAV  automatically 
stabilizes  its  altitude.  While  it  is  recoverable,  depending  on  the  tactical  situation,  the  soldier  may 
discard  this  MAV  due  to  its  low  cost. 

To  better  use  this  technology,  soldiers  will  require  a  suitable  Ground  Control  System  (GCS)  to 
interface  with  the  flying  device  they  are  operating.  The  GCS  must  be  subject  to  stringent 
engineering  criteria,  such  as  the  robustness  to  withstand  extreme  heat,  moisture  and  acceleration. 
The  GCS  must  also  be  small  and  light  enough  to  not  seriously  hinder  the  soldier  who  carries  it. 
To  interact  and  control  a  MAV  through  the  GCS,  the  soldier  requires  a  graphic  user  interface 
(GUI)  embedded  within  the  GCS.  This  GUI  should  be  subject  to  stringent  Human  Factors  (HF) 
engineering  design  criteria,  such  as  the  amount  of  sensory  data  displayed,  screen  size,  resolution, 
and  optimized  map  and  video  sensor  views,  etc.  It  must  be  easy  to  learn  for  the  average  soldier, 
and  must  be  intuitive  in  functionality  and  display  so  that  quick  responses  can  be  achieved  during 
mission  critical  moments  in  deciphering  what  the  next  control  sequence  should  be. 

A  number  of  handheld  devices  have  been  developed  for  Uninhabited  Vehicles  (UVs)  and  robots. 
These  include  Personal  Digital  Assistants  (PDAs)  (Fong,  Thorpe,  &  Glass,  2003;  Huttenrauch  & 
Norman,  2001;  Miller,  et  al.,  2003),  portable  tablet  laptops  (Rutley,  2005),  and  modular  systems 
(Quigley,  Goodrich,  &  Beard,  2004;  Murphy,  Wu,  &  Miller,  2007).  Handheld  devices  may  either 
serve  in  a  larger  GCS  as  a  component  of  the  operator  interface  or  video  interface,  or  they  can 
function  as  the  integrated  GCS.  However,  higher  automation  in  UVs  may  impair  situation 
awareness  (SA)  and  lead  to  accidents  (Parasuraman  &  Miller,  2006;  Miller  &  Parasuraman, 
2007).  Despite  developed  design  principles  for  PC-based  UV  interfaces  (Hou  &  Kobierski,  2006; 
Hou,  Kobierski,  &  Brown,  2007;  Hou,  Kobierski,  &  Herdman,  2006),  insufficient  HF  work  is 
available  to  determine  the  most  appropriate  hardware  platform  and  the  most  effective  interface 
design  method  for  controlling  a  MAV  using  a  handheld  device. 

To  address  the  lack  of  HF  research  on  desirable  interface  design  approach  for  a  handheld  device 
to  control  a  MAV,  Defence  Research  and  Development  of  Canada  (DRDC)  -  Toronto  conducted 
a  HF  study  on  the  design  principles,  implementation,  and  evaluation  of  GUIs  on  different  types  of 
handheld  input  devices  and  interface  design  approaches.  General  usability  heuristics  were  used  in 
this  study.  Research  components  in  this  study  included  several  tasks:  1)  development  of  concept 
of  operations  (CONOPS)  for  a  MAV;  2)  development  of  a  draft  Statement  of  Requirements 
(SOR)  of  a  MAV  system;  3)  a  focus  group  validation  study  of  SOR;  4)  development  of  prototype 
GUIs;  5)  evaluation  of  prototype  GUIs;  and  6)  recommendations  on  GUI  design  principles. 

With  the  developments  of  CONOPS  and  draft  SOR,  identified  characteristics  include:  GCS 
physical  aspects,  MAV  stowage  and  maintenance,  desired  MAV  performance  characteristics,  and 
desired  GUI  features  and  functions  (Ste-Croix  &  Angel,  2008).  The  SOR  set  forth  usage 


DRDC  Toronto  TR  2010-075 


1 


guidelines  for  the  GCS  and  GUI,  but  indicated  that  there  were  many  unanswered  questions 
regarding  the  ideal  platform  and  interactions  to  support  in-theatre  MAV  use.  The  MAV  needs  to 
be  small  enough  to  carry  in  a  backpack  and  inexpensive  enough  to  leave  behind  if  damaged.  A 
lighter  and  smaller  sized  aircraft  and  GCS  will  result  in  higher  mobility  of  the  soldier  carrying  the 
equipment.  A  particular  challenge  was  the  ability  to  fly  through  a  window  or  doorway  to  provide 
intelligence  regarding  a  building’s  contents.  The  focus  group  study  showed  that  gaming-style 
controllers  (Xbox  360)  lead  to  faster  training  and  easier  operation  in  the  field.  Within  this 
category  of  control  system  devices,  two  other  types  of  GUIs  were  developed  on  Sony’s 
PlayStation  Portable  (PSP)  and  Nokia  Internet  Tablet  (Haylock,  2008,  Hou,  et  al.,  2009).  Due  to 
the  incompatibility  of  the  PSP  with  Adobe  Flash™  software  (version  10),  an  Ultra  Mobile 
Personal  Computer  (UMPC)  Viliv  S5  was  identified  and  used  to  develop  GUIs  and  evaluate 
interface  design  concepts.  The  Viliv  S5  is  portable  and  compact,  and  a  5-inch  (diagonal)  touch 
screen.  It  also  has  tactile  buttons  and  a  mouse  for  control  input. 

An  unanswered  question  from  the  focus  group  study  was  what  display  mode  of  a  GUI  can  give  an 
operator  better  SA  given  the  limited  screen  size  of  a  handheld  device.  Would  the  operator  get 
better  SA  with  a  simultaneous  display  having  both  senor  and  map  views  than  with  a  sequential 
display  having  one  view  at  a  time  to  use  the  maximum  screen  size?  Another  unanswered  question 
from  the  focus  group  study  is  what  command  control  input  method  is  more  effective  to  control  a 
MAV?  Would  it  be  more  effective  to  input  control  command  through  touch  screen  than  through 
tactile  buttons  of  the  handheld  device? 

Since  these  two  questions  are  directly  related  to  the  effective  control  of  the  MAV  and  maintaining 
SA  for  the  operator,  an  empirical  study  was  planned  and  performed  to  address  these  questions  as 
the  first  step  of  the  development  of  interface  design  guidelines.  This  report  describes  the 
methodology,  the  design  and  implementation,  the  experimentation,  and  the  research  results.  The 
findings  provide  a  starting  point  in  understanding  how  an  operator  interface  on  a  hand  held  device 
can  be  designed  for  effectively  controlling  a  MAV  system.  The  implications  of  this  study 
contribute  to  the  development  of  SOR  for  MAV  systems. 
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2  Experiment  Platform 


2.1  Apparatus 

The  experiment  platform  was  developed  in  a  number  of  phases,  including  requirement  analysis, 
system  design,  implementation,  and  quality  assurance.  A  complete  experimental  system  consisted 
of  two  standard  IBM  Personal  Computer  (PC),  two  17”  LCD  monitors,  two  standard  US-105  Key 
IBM  keyboards,  two  standard  USB  optical  IBM  mouse,  a  Viliv  S5  Ultra  Mobile  Personal 
Computer  (UMPC)  with  a  screen  size  of  10.5cm  x  6.3cm  and  a  screen  resolution  of  1024  x  600 
pixels,  and  a  Linksys  WRT54GL  wireless  router  to  connect  the  two  computers,  as  shown  in 
Figure  1.  Both  computers  used  the  Windows  XP  Professional  2002  operating  system  displayed  on 
a  LCD  monitor  with  screen  dimensions  41  cm  x  31  cm  and  a  resolution  1280  x  1024  pixels 
resolution  with  the  participant  seated  approximately  55  cm  away  from  the  monitor.  The  virtual 
MAV,  the  virtual  environment,  and  the  user  interface  were  implemented  using  Adobe  Flash™, 
LUA  Code,  and  Virtual  Navigation  and  Collaboration  Experimentation  Platform  (VNCEP) 
(Perlin,  2008).  The  experimental  data  collection  and  pre-process  mechanisms  were  implemented 
using  Visual  Basic  and  C#  programming  languages. 


Figure  1:  The  Experimental  Setup 


The  system  and  GCS  were  designed  and  developed  to  simulate  a  real  world  environment 
including  control  methods  and  sensory  data  display.  For  example,  the  sensory  update  frequency 
was  3-4  Hz  on  the  experimental  setup,  which  is  a  latency  value  that  has  been  demonstrated  not  to 
produce  any  adverse  effects  on  the  controllability  of  the  UV  (Billings  &  Durlach,  2008).  Either  a 
‘GO’  or  ‘STOP’  button  was  displayed  on  the  screen  at  any  given  time  in  Manual  flight  mode  to 
avoid  any  unnecessary  clutter  on  the  graphical  user  interface  (GUI).  Another  feature  was  seen  in 
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autopilot  mode;  while  in  flight,  the  altitude  of  the  MAV  was  set  to  automatically  fly  the  vehicle 
above  the  buildings  within  the  virtual  environment. 

The  GUI  was  developed  using  Adobe  Flash™  version  10.  A  client  and  server  architecture  (Figure 
2)  was  employed.  The  client  ran  the  developed  GUI  on  the  Viliv  S5  handheld  GCS.  The  server 
consisted  of  a  virtual  MAV  environment  and  corresponding  experiment  administration  features. 
These  features  can  monitor  the  mission  and  pause  the  mission  or  stop  the  user  control  at  any  time. 


Client 


-  «»  «.  Request  for  client 


Memory 


Load  into  memory 


Flash  Client 


1 _ 

Sensory  data  requests  _  j 

Sensorv  data  undates 

w 

MAV  system 

w 

Figure  2:  Client-Server  Architecture 


The  following  views  were  incorporated  into  the  interface  for  the  control  of  a  MAV:  System  Status, 
Sensor,  and  Map.  Each  view  provided  different  essential  capabilities  for  the  operator  to  control 
and/or  monitor  the  MAV.  The  Viliv  S5  handheld  device  was  chosen  for  this  study  due  to  its 
portability  and  compactness.  The  device  has  a  5-inch  screen  in  diagonal. 

Figure  3  shows  the  System  Status  view.  The  System  Status  includes  the  status  of  the  GPS  sensor, 
the  Communication  Link,  and  the  MAV  Battery  at  the  time  of  connection  between  GCS  and  the 
MAV.  A  green  check  mark  represents  proper  working  order.  The  actual  GUI  in  the 
Sensor  and  Map  view  is  different  with  respect  to  two  different  display  modes  described  in  detail 
in  Section  2.2. 
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Figure  3:  System  view  displaying  MAV  Status  View  with  interface  features 

Figure  4  shows  an  example  GUI  with  a  simultaneous  display  of  the  MAV  at  the  starting  position. 
The  left  side  of  the  screen  shows  the  map  view  of  the  virtual  environment,  and  the  top  right  side 
of  the  screen  shows  the  sensor  view.  The  display  controls  are  at  the  top  right  comer  of  the  screen. 
Clicking  on  the  Status,  Sensor  or  Map  View  icons  allows  the  user  to  view  the  respective  displays. 
Under  the  icons  are  the  MAV  directional  controls.  ALT  is  the  Altitude  meter  which  gives  the 
user  the  ability  to  adjust  the  altitude  of  the  MAV  by  clicking  along  the  meter  during  the  mission. 
HDG  is  the  Heading  dial  which  allows  the  user  to  rotate  the  MAV  by  clicking  an  angle  within  the 
dial  to  adjust  its  heading  direction.  The  flight  modes  are  featured  at  the  bottom  right  comer  of  the 
screen,  where  the  MAV  flies  automatically  through  designated  waypoints  during  Auto  mode  and 
at  the  command  of  the  user  during  Manual  mode. 


Map  view  Altitude  Meter 


Sensor  view 


Heading  Dial 


Figure  4:  Simultaneous  Display  Mode  shown  on  the  Viliv  S5  UMPC 
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Other  buttons  on  the  screen  include  Reset,  shown  in  Figure  6,  which  resets  the  mission  when 
pressed  and  Create  Route  shown  in  Figure  8,  which  allows  the  user  to  designate  waypoints  for  the 
MAY  to  fly  through. 


2.2  The  Display  Modes  and  Features 

Two  display  modes  were  used  in  the  experiment:  sequential  display  mode  and  simultaneous 
display  mode.  Both  of  these  displays  included  an  Auto  mode  and  a  Manual  mode.  Auto  mode 
allowed  the  vehicle  to  fly  automatically  through  the  set  waypoints  and  Manual  mode  allowed  the 
operator  to  manually  control  its  flight  path.  The  map  and  sensor  views  in  both  display  modes  are 
shown  in  Figure  5  through  Figure  10.  Figure  5  shows  the  sensor  view  in  Manual,  sequential 
display  mode.  The  interface  features  include  Up,  Down,  Left  and  Right  directional  control 
buttons  which  allow  for  precise  adjustment  of  the  MAV’s  position.  Zoom  In  and  Zoom  Out 
buttons  allow  the  users  to  focus  the  sensor  view. 


Left  directional 
Control  Button 


Up  Directional 
Control  .Button 


Right  Directional 
Control  Button 


1 1  Status 

iHh 

[V«. 


Selected 
Sensor  View 


h 


alt 


_Zoom  In 
Button 


Zoom  Out 
Button 


Down  Directional  Control  Button  Selected  Manual  Mode 


Figure  5:  Sensor  View  in  Manual,  Sequential  Display  Mode  with  interface  features 

Figure  6  shows  the  map  view  in  Manual,  sequential  display  mode.  The  Go  Button  enabled  the 
MAV  to  fly  straight  at  a  constant  speed,  and  the  Reset  button  restarts  the  mission  if  failed.  The 
user  was  able  to  designate  waypoints  in  the  Auto  mode  either  by  touching  or  clicking  on  the 
desired  locations,  as  shown  in  Figure  7.  Here,  the  waypoints  are  indicated  with  green  (first 
waypoint),  blue,  and  red  (last  waypoint)  crosses.  The  right  side  features  a  Save  Route  button, 
which  allows  the  user  to  save  the  created  waypoints.  An  Undo  Button  allows  the  user  to  delete  the 
last  waypoint  created.  A  Stop  and  Clear  Button  allows  to  clear  all  the  waypoints  created.  Figure  8 
shows  the  interface  after  setting  the  waypoints  and  saving  them.  The  Start  Mission  button 
commences  the  flight  of  the  MAV  through  the  trajectory  path  represented  by  the  red  lines. 
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Figure  6:  Map  View  in  Manual,  Sequential  Display  Mode  with  Interface  Features 


Save  Route  Button 


Stop  and  Clear 


Figure  7:  Map  View  in  Auto,  Sequential  Display  Mode  with  Interface  Features 
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Create  Route 


Status 


Start  Mission 
Button 


Figure  8:  Map  View  in  Auto,  Sequential  Display  Mode  with  Interface  Features 


The  sensor  view  in  Manual,  simultaneous  display  mode  (Figure  9)  contains  the  same  features  as 
Figure  5,  except  for  the  added  map  view  on  the  top  right  side  of  the  interface. 


Sensor  View 


Figure  9:  Sensor  View  in  Manual,  Simultaneous  Display  Mode  with  Interface  Features 
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Figure  1 0  depicts  the  Map  view  in  Manual,  simultaneous  display  mode,  which  contains  the  same 
features  as  Figure  6  through  Figure  8  but  with  an  added  sensor  view  on  the  top  right  side 
of  the  interface. 


Figure  10:  Map  View  in  Manual,  Simultaneous  Display  Mode 


From  the  above  figures,  it  can  be  seen  that  the  simultaneous  display  mode  was  designed  in  a 
picture-in-picture  style.  The  map  view  of  the  simultaneous  display  mode  features  the  map  as  a 
large  image,  while  the  sensor  is  displayed  as  a  small  image,  and  vice  versa  for  the  sensor  view. 


2.3  The  Control  Input  Type 

Two  command  control  input  types  were  used  in  the  experiment:  touch  screen  and  tactile  buttons, 
as  shown  in  Figure  1 1  and  Figure  12  below.  The  touch  screen  control  requires  that  the  user  uses  a 
stylus  to  point  and  click  on  a  button  on  the  screen,  as  shown  in  Figure  1 1  right.  The  tactile  button 
control  requires  that  the  user  navigates  the  cursor  with  the  mouse  stick  and  select  by  pressing  the 
OK  button  (left  click),  as  shown  in  Figure  1 1  left  and  Figure  12. 
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Figure  11:  The  Tactile  Button  Control  (left)  and  the  Touch  Screen  Control  (right) 
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3  Experimental  Design 


To  determine  effective  handheld  GUI  display  mode  and  device  input  methods,  a  2  x  2  factorial 
and  within  subjects  design  was  used  in  the  experiment.  The  experiment  also  looked  into 
correlation  between  PC  and  handheld  device  when  they  were  used  for  training  as  well  as  their 
effects  on  all  six  dependent  variables. 

3.1  Participants 

Forty  four  (44)  volunteers  from  DRDC  Toronto  and  the  surrounding  community  were  recruited 
by  email  to  participate  in  the  experiment.  They  were  aged  18-65  (M=28.16,  SD=7.24),  with 
normal  or  corrected-to-normal  vision.  None  of  them  had  experience  controlling  a  MAV  before. 
Participants  were  to  fill  out  a  background  questionnaire  (Annex  D)  and  were  financially 
compensated  for  stress  allowance. 

3.2  Independent  Variables 

The  effectiveness  of  the  GUI  interface  design  on  aiding  operators’  performance  was  examined 
using  a  2  (display  mode  condition:  simultaneous  view  versus  sequential  view)  x  2  (control  input 
type:  touch  screen  versus  tactile  button)  repeated  measures  in  design.  There  were  two  trials  for 
each  of  the  four  experimental  conditions  created  by  factorially  crossing  these  two  variables.  The 
experiment  conditions  are  shown  in  Table  1,  where  touch  screen  is  represented  by  To,  tactile 
button  is  represented  by  Ta,  simultaneous  display  is  represented  by  Si,  and  sequential  display  is 
represented  by  Se. 


Table  1:  Experimental  conditions  for  MAV  study. 


Touch  Screen  (To) 

Tactile  Button  (Ta) 

Simultaneous  (Si) 

Trial  To-Si 

Trial  Ta-Si 

Sequential  (Se) 

Trial  To-Se 

Trial  Ta-Se 

3.3  Dependent  Variables 

Both  objective  and  subject  measures  were  used  to  index  each  participant’s  performance  under  two 
display  modes  and  two  control  input  types.  There  are  five  objective  measures  for  the  experiment: 
(1)  trajectory  error  of  the  flight;  (2)  task  completion  time;  (3)  situation  awareness  using  situation 
awareness  global  assessment  technique  (SAGAT);  (4)  display  switch  frequency  (between  sensor 
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view  and  map  view);  and  (5)  training  time.  There  was  only  one  subjective  measure  which  is  (6) 
mental  workload  using  the  NASA  TLX. 

3.3.1  Trajectory  Error 

Trajectory  error  to  each  defined  waypoint  (see  Figure  13)  was  measured  using  the  closest  distance 
of  the  MAV  trajectory  from  that  waypoint.  Waypoints  1,  2,  and  3  in  Figure  13  were  recorded 
only  in  2  dimensions  (X  and  Y  co-ordinates)  as  a  specific  altitude  (Z  co-ordinate)  of  flight  was 
not  required  for  these  waypoints.  Waypoints  A  and  E  which  are  building  access  and  exit 
windows  were  recorded  in  3D  (X,  Y,  and  Z  co-ordinates)  as  all  the  experiment  required  the 
windows  be  traversed  in  the  center  including  a  specific  altitude.  The  trajectory  error  of  a  trial  is 
calculated  as  the  average  trajectory  error  over  all  the  5  waypoints. 


Figure  13:  Map  with  defined  waypoints  (red  dot  represents  NPTC  in  room) 


3.3.2  Task  Completion  Time 

The  time  recorded  from  the  start  (waypoint  0  in  Figure  13)  to  the  end  of  the  mission  (waypoint  E) 
was  the  completion  time.  Due  to  the  fact  that  both  Auto  and  Manual  modes  were  used  within  a 
single  mission,  the  task  completion  time  included  the  time  used  in  both  modes. 

3.3.3  Situation  Awareness 

Situation  awareness  was  measured  using  the  SAG  AT,  which  provides  an  objective  measure  of  SA 
based  on  queries  during  freezes  of  a  simulation  (Endsley,  1988).  During  the  simulated 
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experimental  task,  the  mission  was  paused  and  the  participant  was  given  a  questionnaire  (see 
Annex  F)  on  the  location  (both  Map  view  and  Sensory  view)  of  the  MAV  at  that  particular  pause 
point.  The  participant’s  answer  was  compared  with  the  ground  truth  that  was  simultaneously 
collected  in  the  computer  database.  The  comparison  of  the  ground  truth  and  perceived  SA  was 
used  as  a  dependent  variable  in  this  study  to  provide  an  objective  measure  of  situation  awareness. 
The  SAGAT  questionnaire  was  administered  after  the  first  trial  of  the  first  simultaneous  display 
mode  and  first  sequential  display  mode  for  a  total  of  two  questionnaires  on  each  participant. 

3.3.4  Mental  Workload 

Mental  workload  for  each  experiment  condition  was  gathered  by  administering  a  NASA  TLX 
mental  workload  questionnaire  (Annex  E)  developed  by  Hart  and  Staveland  (1988).  TLX  focuses 
on  six  factors  contributing  to  the  workload  as  a  pair-wise  comparison:  Mental  Demand  (MD), 
Physical  Demand  (PD),  Temporal  Demand  (TD),  Effort  (EF),  Performance  (PF),  and  Frustration 
(FR).  Since  the  primary  interest  of  this  study  were  the  effort  and  ease  of  use  of  different  display 
modes  and  command  input  methods  when  controlling  a  MAV,  the  workload  measure  focused  on 
effort  and  temporal  demand  rather  than  performance  which  was  evaluated  by  other  objective 
measures.  The  “importance”  list  of  six  factors  is  illustrated  in  Table  2  below.  With  the  list,  the 
number  marked  as  importance  was  counted  and  used  as  a  weight  for  each  factor.  The  weight 
chart  is  shown  in  the  right  column  of  Table  3  in  which  effort  was  given  the  highest  weight  (5)  and 
performance  was  given  the  lowest  weight  (0).  A  rating  scale  (from  0  to  10)  on  each  factor  was 
provided  to  each  participant.  The  mental  workload  for  a  subject  was  recorded  as  the  sum  of  the 
weighted  scale,  which  was  between  0  and  15.  The  questionnaire  was  administered  after  the 
second  repetition  of  each  trial  condition  for  a  total  of  5  questionnaires  for  each  participant. 

3.3.5  Display  Switch  Frequency 

During  the  experiment,  the  participants  were  able  to  click  an  on  screen  button  to  switch  display 
modes,  either  from  map  view  to  sensor  view,  or  vice  versa.  The  number  of  such  button  clicks 
was  recorded  into  a  database  as  display  switch  frequency.  This  frequency  was  used  to  determine 
how  often  a  participant  was  switching  views  to  compare  the  effectiveness  of  simultaneous  and 
sequential  display  modes. 

3.3.6  Training  Time 

The  training  session  at  the  beginning  of  the  experiment  was  intended  to  reduce  a  participant’s 
performance  variation  caused  by  different  levels  of  experience  with  touch  screen  and/or  tactile 
button  devices.  Each  participant  was  required  to  reach  a  pre-defmed  level  of  experience  in  each 
experimental  condition  before  successful  completion  of  the  training  session.  They  needed  to  fly 
the  MAV  using  Viliv  S5  from  the  start  point  (waypoint  0  in  Figure  13)  to  the  exit  of  the  simulated 
building  (waypoint  E  in  Figure  13)  without  any  incidents  such  as  wall  collisions  or  flying  off  the 
map.  The  time  that  a  participant  used  during  each  condition  in  training  on  Viliv  S5  was 
recorded  as  a  dependent  variable  of  training  time.  This  was  used  to  evaluate  the  workload  based 
on  training. 
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Table  3:  NASA  TLX  item  weight  chart 


Mental  Demand 

3 

Physical  Demand 

1 

Temporal  Demand 

4 

Effort 

5 

Performance 

0 

Frustration 

2 

Total  Weight 

15 

3.4  The  Tasks 

3.4.1  The  Baseline  Walking  Task 

The  walking  task  was  conducted  in  the  training  session  on  a  PC  with  a  1 7”  LCD  monitor  (display 
characteristics  are  described  in  section  2.1)  for  participants  to  get  familiar  with  the  experimental 
environment.  It  involved  the  following  steps: 

1.  From  starting  location  0  as  indicated  in  Figure  13, 

2.  Navigate  the  playable  character  to  the  North  West  towards  waypoint  1  on  the  road, 

3.  Follow  given  waypoints  2  and  3  as  shown  in  Figure  13, 

a.  Go  south  on  the  road  until  a  bus  is  seen, 

b.  Locate  the  building  to  the  East  of  the  bus  with  an  open  door,  and 

c.  Enter  the  building  through  this  door. 

4.  Locate  and  carefully  make  your  way  up  the  staircase  inside  the  building,  and 

5.  Locate  the  Non-Playable  Target  Character  (NPTC;  represented  by  red  dot  in  Figure  13)  and 
step  on  the  crate  to  the  right  of  the  NPTC  seen  in  Figure  14. 
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Figure  14:  Non-Playable  Target  Character  (centre)  and  crate  (right) 


3.4.2  The  MAV  Fly  Task 

This  task  was  conducted  on  both  PC  and  Viliv  S5  for  both  training  and  experimental  sessions.  It 
has  the  following  six  steps: 

1.  From  starting  location  0  as  indicated  in  Figure  13  (Auto  mode), 

2.  Fly  through  the  predefined  way-markers  1,  2  and  3  in  the  map  view  (Auto  mode), 

3.  Turn  heading  dial  towards  East  to  face  the  predefined  building,  and  lower  the  altitude  to  6.2 
m  AGL  (At  Ground  Level)  (Manual  mode), 

4.  Locate  building  to  the  East  of  the  bus  and  fly  through  window  A  as  shown  in  Figure  13 
(Manual  mode), 

5.  Identify  the  NPTC  in  the  room  (Manual  mode),  and 

6.  Fly  out  of  the  room  through  window  E  as  shown  in  Figure  13  (Manual  mode). 

The  MAV  fly  task  was  completed  with  both  Auto  mode  and  Manual  mode.  The  experiment  data 
were  collected  and  analyzed  in  terms  of  tasks  completed  in  both  Auto  mode  and  Manual  mode. 

3.5  Counterbalancing  with  Trial  Order 

Since  a  2  x  2  factorial  design  was  used  in  this  experiment,  experimental  conditions  were 
counterbalanced  across  participants  to  prevent  learning  effects  from  contaminating  the  data.  That 
is  half  of  the  participants  received  the  simultaneous  display  mode  condition  first  and  sequential 
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display  condition  second.  This  order  was  reversed  for  the  other  half  of  the  participants.  Control 
input  condition  was  also  counterbalanced  across  all  participants.  That  is,  half  of  the  participants 
received  the  touch  screen  condition  first  and  the  tactile  button  condition  second.  This  order  was 
reversed  for  the  other  half  of  the  participants.  Thus,  participants  were  required  to  be  randomly 
divided  to  form  four  groups  for  a  completed  balanced  design,  as  shown  in  Table  4.  Each  group 
had  1 1  participants  who  had  the  same  experimental  trial  orders. 


Table  4:  Experimental  trial  order  for  each  group 


Group 

Number 

Trial  1 
(Handheld) 

Trial  2 
(Handheld) 

Trial  3 
(Handheld) 

Trial  4 
(Handheld) 

+ 

Questionnaires 

1 

To-Si 

Ta-Si 

To-Se 

Ta-Se 

2 

Ta-Si 

To-Se 

Ta-Se 

To-Si 

3 

To-Se 

Ta-Se 

To-Si 

Ta-Si 

4 

Ta-Se 

To-Si 

Ta-Si 

To-Se 

3.6  The  Procedure 

The  experiment  consisted  of  two  separate  sessions  for  each  participant,  and  all  were  conducted  in 
succession  within  a  day.  The  first  session  was  for  training  which  lasted  about  60  minutes.  The 
following  experiment  session  had  four  trials  which  lasted  approximately  15  minutes  each  with 
short  breaks  in-between.  It  was  then  followed  by  a  15  to  20  minute  question  and  debriefing 
period.  The  total  allotted  time  was  about  3  hours,  including  the  breaks. 

The  training  session  was  conducted  on  two  PCs  and  one  Viliv  S5  handheld  device.  This  session 
had  seven  parts  and  the  part  orders  were  specified  as  follows: 

Part  1 :  Environment  familiarization  and  training  for  the  baseline  walking  task  PC  1 ; 

Part  2:  GUI  familiarization  training  for  the  MAV  fly  task  on  PC  2; 

Part  3:  Short  5-minute  break; 

Part  4:  Training  for  Trial  1  -  MAV  fly  task  on  Viliv  S5  (see  Table  4  for  group  order); 

Part  5:  Training  for  Trial  2  -  MAV  fly  task  on  Viliv  S5  (see  Table  4  for  group  order); 

Part  6:  Training  for  Trial  3  -  MAV  fly  task  on  Viliv  S5  (see  Table  4  for  group  order); 

Part  7:  Training  for  Trial  4  -  MAV  fly  task  on  Viliv  S5  (see  Table  4  for  group  order). 

In  Part  1,  participants  controlled  the  playable  character  on  PC  1  in  the  3D  virtual  environment 
using  a  keyboard  and  mouse  in  order  to  get  familiar  with  the  mission.  The  mouse  was  used  to 
control  the  gaze  direction  of  the  player.  The  A,  W,  S,  and  D  keys  on  the  keyboard  were  used  to 
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control  the  movement  of  the  player.  In  Part  2,  participants  used  PC  2  to  get  familiar  with  two 
GUI  display  conditions  (simultaneous  display  and  sequential  display),  the  functionality  of  the 
system,  and  the  experimental  MAV  fly  task  (see  section  3.4.2).  PC  2  was  used  to  replicate  the 
connection  to  server  method  from  each  GUI  to  control  the  MAV.  From  Part  4  through  7,  the 
handheld  device  training  was  executed  on  the  Viliv  S5  in  a  specified  order  based  on  a 
participant’s  group  number  for  experimental  trials  (as  shown  in  Table  4).  Each  of  these  four  parts 
was  timed  individually  from  the  moment  the  training  began  until  the  moment  the  fly  task  was 
successfully  completed  without  any  incidents  such  as  wall  collisions  or  flying  off  the  map.  This 
process  familiarized  participants  with  the  operational  environment,  the  different  control  types  and 
display  modes,  as  well  as  the  task.  The  training  times  were  recorded  from  the  PC  baseline 
walking  task  for  the  playable  character  and  handheld  training  with  each  condition  and 
used  for  learning  effect  analysis  purposes  and  to  examine  for  correlation  between  PC  and 
handheld  task  training. 

After  completing  the  training  session,  participants  conducted  the  experimental  session  including 
four  MAV  fly  task  trials  on  Viliv  S5.  The  order  of  trials  was  based  on  individual  participants’ 
group  number  as  shown  in  Table  4.  Participants  repeated  each  trial  twice  in  each  condition. 

3.7  Data  Collection  and  Verification 

The  experiment  data  including  the  playable  character  data,  MAV  data,  all  the  events  from  client 
and  server  during  training  and  experiment  sessions  were  recorded  automatically  and  manually. 
The  data  logging  module  within  the  server  logged  all  the  data  to  a  text  file.  Then  this  text  file  was 
organized  and  exported  into  a  single  Microsoft  Office  Excel  sheet  for  further  analyses. 

Six  dependent  variables  were  recorded  during  the  experiment.  Four  of  the  six  were  automatically 
recorded  by  the  computer  throughout  the  experiment.  They  were  (1)  trajectory  error,  (2)  task 
completion  time,  (3)  display  switch  frequency,  and  (4)  training  time.  The  two  other 
variables  that  were  manually  recorded  and  calculated  based  on  questionnaires  were  (1)  NASA 
TLX  and  (2)  SAGAT. 

To  facilitate  the  registration  of  a  data  set  to  the  corresponding  trial,  the  recorded  playable 
character  or  MAV  trajectory  was  plotted  on  the  map.  These  plotted  trajectories  allowed  the 
experimenter  to  visually  verify  whether  the  recorded  data  had  any  sporadic  errors  when  the  data 
were  logged  or  when  it  were  imported  into  the  Excel  sheet  for  analysis. 

Figure  15  below  shows  the  visualized  trajectory  of  the  playable  character  from  an  experimental 
trial.  The  yellow  cross  indicates  the  defined  waypoints.  The  yellow  circles  show  the  name  of 
each  waypoint.  The  dotted  blue  line  shows  the  trajectory  of  the  playable  character. 
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Figure  15:  A  visual  trajectory >  from  the  playable  character  data  record 


Figure  16  shows  the  flight  path  from  an  experimental  trial.  The  dotted  yellow  line  shows  the 
MAV  route  travelled.  The  blue  crosses  show  the  waypoints  and  the  blue  circles  show  the  name 
of  the  waypoints.  The  white  MAVs  show  the  closest  position  and  heading  relative  to  the  defined 
waypoints.  The  blue  MAV  over  the  green  shows  the  position  and  heading  when  the  system  was 
paused  for  the  SAGAT  questionnaire. 
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Figure  16:  A  visual  trajectory  from  the  MAV  data  record 
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4  Experimental  Results 


4.1  Analytical  Method 

After  the  data  were  retrieved,  the  Analysis  of  Variance  (ANOVA)  was  performed  on  each 
dependent  variable  with  respect  to  two  factors:  display  mode  and  control  input  type.  To  examine 
the  correlation  between  PC  and  handheld  devices  for  training  and  their  effects  on  all  six 
measures,  a  multivariate  analysis  of  covariance  (MANCOVA)  was  also  used  for  covariate 
analysis.  The  software  STATISTICA™  version  9  was  used  to  conduct  the  analyses.  The  results 
and  associated  95%  confidence  intervals  for  both  display  mode  and  control  type  factors  are 
presented  in  the  following  sections  for  six  dependent  variables. 

4.2  Trajectory  Error 

The  mean  trajectory  error  as  a  function  of  control  type  is  shown  in  Figure  17.  Control  type  had  a 
significant  effect  on  trajectory  error,  F  (1,  173)  =  4.35,  p  <  .05.  The  touch  screen  input  method 
yielded  larger  trajectory  errors  (55.99  cm)  than  the  tactile  button  input  (50.49  cm).  There  were  no 
other  significant  effects,  p  >  .05  and  there  was  no  interaction  between  display  mode  and  control 
type,  p  =  0.28. 


Mean  Trajectory  Error  as  a  Function  of  Control  T>pe 
Error  Bar  =  95%  CL  F(L  173)=4.35,  p<  05 
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Figure  17:  Trajectory  error  plot 
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4.3  Task  Completion  Time 

The  task  completion  time  as  a  function  of  control  type  is  shown  in  Figure  18.  Control  type  had  a 
significant  effect  on  the  time  to  complete  the  task,  F  (1,  173)  =  32.24,  p  <  .05.  The  touch  screen 
yielded  a  shorter  task  completion  time  (139  s)  than  tactile  buttons  (166  s).  There  were  no  other 
significant  effects,  p  >  .05. 


Mean  Task  Completion  Time  as  a  Function  of  Control  T>pe 
Error  Bar  =95%  CL  F(T  173)=32.24,  p<.05 


Figure  18:  Task  completion  time  plot 


4.4  Situation  Awareness 

SAGAT  data  as  a  function  of  display  mode  and  control  type  are  shown  Figure  19  and 
Figure  20.  Both  display  mode  and  control  type  had  significant  effects  on  SAGAT,  with 
F  (1,  85)  =  9.72,  p  <  .05  for  display  mode  and  F  (1,  85)  =  9.77,  p  <  .05  for  control  type. 
Simultaneous  display  led  to  higher  SA  ratings  than  sequential  display.  The  touch  screen  control 
type  also  led  to  higher  SA  ratings  than  tactile  buttons. 
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Situation  Awareness  (SAG AT) 


Situation  Awareness  as  a  Function  of  Display  Mode 

Error  Bar  =  95%  CL  F(l,  85)=9.72,  p<05 
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Figure  19:  Situation  awareness  SAG  AT  plot  for  display  mode 


Situation  Awareness  as  a  Function  of  Control  Type 
Error  Bar  =  95%  CL  F(l,  85)=9.77,  p<.05 
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Figure  20:  Situation  awareness  SAGAT  plot  for  control  type 


DRDC  Toronto  TR  2010-075 


23 


4.5  Mental  Workload 


Mental  workload  as  a  function  of  display  mode  and  control  type  is  shown  in  Figure  21.  Control 
type  had  a  significant  effect  on  mental  workload,  F  (1,  173)  =  14.16,  p  <  .05.  The  perceived 
mental  workload  was  lower  for  the  touch  screen  condition  than  the  tactile  button  condition. 


Mental  Workload  as  a  Function  of  Control  Type 

Error  Bar  =  95%  Cl,  Kl,  173)= 14. 16,  p<.05 


Figure  21:  Mental  workload  plot 


4.6  Display  Switch  Frequency 

Display  switch  frequency  as  a  function  of  display  mode  and  control  type  is  shown  in  Figure  22. 
Display  mode  had  a  significant  effect  on  display  switch  frequency,  F  (1,  173)  =  23.69,  p  <  .05. 
Simultaneous  display  mode  yielded  a  significantly  lower  display  switch  frequency  (2.9  switches) 
than  sequential  display  mode  (4.8  switches).  Control  type  also  had  a  significant  effect  on  display 
switch  frequency,  F  (1,  173)  =  5.00,  p  <  .05.  The  touch  screen  control  type  yielded  more  display 
switch  (4.3  switches)  than  the  tactile  button  control  type  (3.39  switches).  There  was  an 
interaction  between  display  mode  and  control  type,  F  (1,  172)  =  5.40,  p  <  .05.  Control  input  type 
made  a  difference  only  for  the  sequential  display  condition,  where  the  touch  screen  (5.75  switches) 
generated  more  switch  frequency  than  tactile  buttons  (3.96  switches). 
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Figure  22:  Display  Switch  Frequency  Plot 


4.7  Training  Time 

Training  time  on  handheld  device  as  a  function  of  display  mode  and  control  type  is  shown  in 
Figure  23.  Control  type  had  a  significant  effect  on  training  time,  F  (1,173)  =  3.96,  p  <  .05.  The 
touch  screen  control  type  yielded  a  shorter  training  time  (202.58  s)  than  tactile  buttons  (229.78  s). 
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Mean  Training  Time  as  a  Function  of  Control  Type 
Error  Bar  =  95%  CL  F(l,  173)=3.99,  p<.05 
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Figure  23:  Training  Time  Plot 

4.8  PC  Baseline  Walking  Training  and  Handheld  Task 

The  training  times  were  recorded  from  the  PC  baseline  walking  task  for  the  playable  character 
and  the  handheld  fly  task  with  each  condition.  The  ANCOVA  analyses  were  conducted  using 
display  mode  and  control  type  as  independent  variables,  training  time  on  PC  baseline  walking 
task  as  a  covariate,  and  each  of  the  six  measures  as  dependent  variables.  The  analyses  produced 
the  similar  result  as  the  ANOVA  analyses  without  PC  baseline  training.  The  MANCOVA 
analysis  was  also  conducted  using  display  mode  and  control  type  as  independent  variables, 
training  time  on  PC  baseline  walking  task  as  a  covariate,  and  all  six  measures  (trajectory  error, 
task  completion  time,  situation  awareness,  mental  workload,  display  switch  frequency  and 
training  time)  as  dependent  variables.  The  analysis  showed  no  significant  effect. 
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5  Observations  and  Discussions 


The  results  of  the  experiment  provided  empirical  evidence  for  the  development  of  an  effective 
GCS  and  could  contribute  to  a  SOR  for  the  control  of  MAV  systems.  The  observations  and 
discussions  are  presented  here. 

5.1  Display  Mode 

The  basic  task  of  this  study  for  participants  is  to  remotely  navigate  a  MAV  through  waypoints  to 
fly  into  a  window  of  a  building  and  fly  out.  Participants  needed  to  plan  waypoints  and  find 
optimal  entry  and  exit  positions  of  the  building  to  avoid  collisions.  This  nature  of  the  MAV 
control  needs  to  have  both  global  (map)  and  local  (sensor)  views  provided  by  a  GUI  though  these 
two  views  may  not  be  needed  simultaneously  all  the  time.  With  the  sequential  display,  there  was 
only  one  map  or  sensor  view  in  the  display.  Participants  had  to  click  the  sensor  or  map  view  icon 
on  the  upper  right  comer  of  the  screen  to  get  the  map  or  sensor  view.  They  had  to  switch  back 
and  forth  between  map  view  and  sensor  view  to  be  fully  aware  of  both  global  and  local  situations 
to  accomplish  the  tasks.  With  the  simultaneous  display,  both  sensor  and  map  views  were  on  one 
display.  Both  global  and  local  information  was  provided  although  one  view  at  the  top  was 
smaller  than  the  other.  It  allowed  participants  to  be  aware  of  the  situation  without  time  delay  or 
visual  feedback  (display)  lag.  This  is  likely  the  reason  why  the  simultaneous  display  mode 
provided  significantly  better  SA  than  the  sequential  display  mode,  as  measured  with  the  SAGAT. 

Again,  both  map  and  sensor  views  were  needed  for  planning  waypoints  and  entry  and  exit 
positions  through  building  widows  for  the  MAV.  Having  both  map  and  sensor  views  on  one 
display,  the  simultaneous  display  provided  both  global  and  local  information.  Participants  did  not 
need  to  switch  back  and  forth  between  the  map  and  sensor  view  unless  they  required  a  large 
specific  view.  For  example,  they  might  have  to  switch  from  a  map  view  to  a  sensor  view  to  get 
detailed  local  information  when  they  were  planning  to  fly  into  and  fly  out  the  building  window. 
They  might  not  need  to  always  switch  from  a  sensor  view  to  a  map  view  which  could  still  provide 
global  information  without  details  due  to  the  small  window.  However,  the  sequential  display 
which  had  only  one  view  at  a  time.  Participants  had  to  switch  back  and  forth  between  map  view 
and  sensor  view  to  have  both  global  and  local  information.  This  perhaps  is  the  reason 
why  the  simultaneous  display  mode  provided  a  lower  display  switch  frequency  than  the 
sequential  display  mode. 

After  having  repeated  the  mission  many  times  during  the  training  session,  the  participant  was 
familiar  with  the  task,  and  may  not  have  found  difficulties  in  the  task  itself  between  the 
simultaneous  and  sequential  displays.  This  probably  is  why  the  perceived  mental  workload, 
which  was  recorded  after  the  second  repetition  of  each  trial  condition,  was  very  similar 
between  simultaneous  and  sequential  display  modes.  This  may  be  the  reason  why  trajectory 
error,  task  completion  time,  and  training  time  were  also  similar  between  simultaneous  and 
sequential  displays. 

The  de-briefing  questionnaire  showed  that  all  participants  preferred  the  simultaneous  display 
mode  over  the  sequential  display  mode. 
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5.2  Control  Type 


The  experiment  results  revealed  that  the  touch  screen  control  type  was  the  overall  preferred 
command  input  method.  Based  on  the  observations  in  the  experiment,  participants  found  it 
frustrating  to  use  the  tactile  buttons  because  it  was  difficult  and  time  consuming  to  navigate  the 
cursor  to  the  next  command.  If  participants  were  not  cautious  when  moving  the  cursor,  it  would 
overshoot  the  command,  which  impeded  the  accuracy  and  speed  of  the  mission.  This  is  likely  the 
reason  why  participants  needed  less  training  time  when  using  touch  screen  than  using  tactile 
buttons.  Perhaps  for  the  same  reason,  participants  produced  lower  mental  workload  scores  when 
using  the  touch  screen  than  when  using  tactile  buttons.  Under  less  mental  workload,  participants 
had  better  SA  when  using  touch  screen  than  using  tactile  buttons. 

With  the  touch  screen,  it  needed  only  one  click  on  a  function  icon  on  the  display  when  selecting 
the  function  using  a  stylus  (e.  g.,  switch  back  and  forth  between  map  and  sensor  views). 
However,  with  tactile  buttons,  participants  had  to  control  a  tactile  mouse  on  the  handheld  device 
and  move  the  cursor  to  the  desired  function  icon  on  the  display  and  then  press  a  button  to  select 
the  function.  This  is  likely  the  reason  why  task  completion  time  was  shorter  using  the  touch 
screen  control  than  using  tactile  buttons.  Sometimes  participants  were  frustrated  with  the 
difficulty  to  control  small  tactile  buttons  on  the  device  and  gave  up  the  control.  The  de-briefing 
questionnaire  showed  that  all  participants  preferred  the  touch  screen  over  tactile  buttons.  It  might 
be  the  reason  why  the  touch  screen  generated  more  display  switches  than  tactile  buttons. 

The  wide  tip  of  the  pointing  device  used  as  the  touch  screen  control  type  (stylus)  might  have 
made  the  control  less  accurate.  It  might  the  reason  why  the  touch  screen  yielded  larger  trajectory 
errors  than  tactile  buttons.  Thus  there  was  a  speed-accuracy  trade-off  for  the  touch  screen 
command  input  method. 

5.3  Interaction  Between  Display  Mode  and  Control  Type 

There  was  an  interaction  between  display  mode  and  control  type  for  the  measure  of  display 
switch  frequency.  Control  input  type  made  a  difference  only  for  the  sequential  display  condition, 
where  the  touch  screen  generated  a  higher  switch  frequency  than  tactile  buttons.  To  navigate  the 
MAV  through  the  building  blocks  and  flying  through  windows  needed  to  have  both  map  and 
sensor  information.  In  the  simultaneous  display  mode,  one  display  provided  both  global  (map) 
and  local  (sensor)  information.  However,  in  the  sequential  display  mode,  there  was  only  one 
view  on  one  display.  It  was  needed  to  switch  back  and  forth  between  map  view  and  sensor  view 
to  get  more  information.  Thus  the  switch  frequency  in  the  sequential  display  mode  was  higher 
than  the  simultaneous  display  for  both  the  touch  screen  and  tactile  buttons.  With  the  touch  screen, 
participants  could  just  make  one  click  using  the  stylus  on  the  upper  right  comer  icon  to  switch 
between  map  and  sensor  views.  But  using  tactile  buttons,  participants  had  to  control  the  mouse  to 
move  the  cursor  onto  the  upper  right  comer  icon  and  press  another  button  to  select  the  view  to 
switch.  Due  to  the  difficulty  to  control  small  tactile  buttons  to  select  functions,  they  might  be 
reluctant  to  use  them  as  often  as  using  a  stylus  to  click  on  the  display.  This  might  be  the  reason 
why  participants  generated  a  lower  switch  frequency  using  tactile  buttons  than  using  the  touch 
screen.  Therefore,  there  existed  an  interaction  between  display  mode  and  control  type. 
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5.4  PC  Baseline  Walking  Task  and  Skill  Compensation 

The  introduction  of  PC  baseline  walking  task  was  to  compensate  participants’  different  PC  skills. 
Another  purpose  was  to  investigate  whether  there  was  a  correlation  between  participants’  PC 
skills  and  their  experiment  performance  on  handheld  devices.  However,  the  experiment  was 
designed  in  such  a  way  that  each  participant  was  tested  on  all  four  experiment  trial  conditions  so 
that  his/her  PC  skills  would  be  contributed  to  each  trial  condition.  Perhaps  this  is  the  reason  why 
both  the  ANCOVA  and  MANCOVA  results  revealed  no  effect  of  training  time  on  PC  baseline 
walking  task  on  the  outcomes  of  the  experiment.  In  the  other  words,  the  experiment  procedure 
might  have  compensated  the  participants’  different  PC  skills. 

5.5  Feedback 

Most  participants  preferred  the  touch  screen  and  simultaneous  display  over  tactile  buttons  and 
sequential  display.  However,  they  also  indicated  that  the  touch  screen  input  method  was  not  as 
precise  when  plotting  the  waypoints.  Some  participants  preferred  using  the  tactile  buttons  when 
creating  waypoints.  On  the  other  hand,  most  participants  preferred  to  use  the  touch  screen  during 
manual  flight  since  it  was  too  time  consuming  to  move  the  cursor  and  press  to  confirm  the  choice 
with  tactile  buttons.  A  couple  of  participants  also  pointed  out  that  from  the  practical  perspective, 
the  touch  screen  would  be  difficult  to  use  when  wearing  a  pair  of  gloves  in  the  field. 
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6  Conclusions 


The  purpose  of  the  study  was  to  provide  guidance  for  the  future  development  for  GCS  interface 
design.  The  experimental  results  revealed  that  the  display  mode  and  control  type  had  significant 
effects  on  the  operation  of  a  MAV  using  a  handheld  device.  The  simultaneous  display  provided 
better  SA  and  fewer  display  switches  compared  with  the  sequential  display.  The  touch  screen 
produced  a  quicker  task  completion  time  and  larger  trajectory  error  than  tactile  buttons,  resulting 
in  a  speed-accuracy  trade-off.  The  touch  screen  also  facilitated  a  shorter  training  time  than  tactile 
buttons.  Additionally,  participants  reported  better  SA  and  lower  mental  workload  using  the  touch 
screen  than  using  tactile  buttons. 

The  research  findings  provided  guidance  for  user  interface  design  of  a  GCS  on  a  handheld  device. 
For  better  SA,  the  interface  should  be  designed  with  a  combination  of  both  sensor  and  map  views. 
For  the  ease-of-control,  a  handheld  device  with  a  touch  screen  is  preferable  over  a  device  with 
tactile  buttons.  These  findings  provided  empirical  evidence  for  the  development  of  SOR  of  MAV 
systems,  but  require  validations  from  field  studies. 
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Annex  A  Experimental  Protocol 


February  3,  201 1 
Protocol:  #L-680 

Title:  Comparison  between  Touch  Screen  and  Tactile  Interfaces  on  Performance  of  Virtual 
Unmanned  Aerial  Vehicle  Operation 

Principal  Investigator:  Dr.  Ming  Hou  (DRDC  Toronto) 

Research  Assistants:  Sheila  Young  and  Andrew  Sun 
Thrust  Code:  12PN03 

A.1  Executive  Summary 

Extremely  Agile  Micro  Aerial  Vehicles  (EA-MAV s)  are  miniature  flying  machines  that  will 
provide  “over-the-hill”  or  “around-the-next-house”  reconnaissance  to  smaller  combat  elements 
such  as  infantry  sections.  With  the  advancement  of  this  technology,  it  is  essential  for  the 
development  of  a  suitable  Ground  Control  System  (GCS)  to  control  it.  Consequently,  there  is  a 
requirement  for  human  factors  research  to  be  done  on  the  subject  to  help  develop  and  deliver  the 
most  efficient  user  interface.  One  significant  factor  to  consider  is  the  differences  between  the  two 
modern  types  of  handheld  input  devices  -  touch  screen  and  tactile  button  equipped  devices.  A 
second  significant  factor  to  consider  is  the  difference  between  a  graphical  user  interface  (GUI) 
that  combines  map  and  video  sensor  data  into  a  single  window  or  one  that  divides  them  into 
separate  windows.  This  study  seeks  to  identify  the  differences  in  performance  and  efficiency  of 
the  operator  when  operating  an  EA-MAV  with  each  of  these  two  types  of  interfaces.  The  results 
will  aid  in  the  design  of  the  final  EA-MAV  GCS  interface  device. 

The  experiment  will  be  conducted  with  the  use  of  the  Virtual  Navigation  and  Collaboration 
Experimentation  Platform  (VNCEP)  developed  by  Esterline  CMC,  a  virtual  environment  with  a 
virtual  EA-MAV  and  a  GUI  developed  by  Vislmage  Systems  Inc.,  and  a  single  handheld  device; 
the  Viliv  S5  Ultra  Mobile  Personal  Computer  (UMPC).  The  participants  will  take  the  role  of  a 
soldier  operating  an  EA-MAV  through  a  simulated  interface.  They  will  fly  through  a  virtual  urban 
terrain  and  into  a  building  with  full  control  over  the  EA-MAV  using  the  Viliv  S5. 

The  experiment  conforms  to  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto 
guidelines  with  respect  to  selection,  payment,  and  treatment  of  human  participants.  Possible  risks 
to  the  subject  are  minimal  and  may  include  mild  fatigue  or  eyestrain  from  performing  the  task 
and/or  using  the  computer  or  a  handheld  device.  The  nature  of  the  experiment,  including  potential 
risks,  will  be  explained  in  detail  to  volunteer  participants  who  will  read  and  sign  a  form  to 
indicate  informed  consent.  The  differences  in  efficiency  and  performance  when  using  touch 
screen  vs.  traditional  tactile  controllers  and  when  using  different  GUI  views  have  not  been 
examined  extensively  in  this  application,  and  the  findings  will  provide  a  starting  point  in 
understanding  how  EA-MAV  GCS  can  be  optimized.  More  specifically,  they  will  provide 
valuable  information  towards  the  final  design  and  construction  of  the  GCS  interface  for  the  EA- 
MAV  project. 
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DRDC  Experiment  Protocol 

Title:  Comparison  between  Touch  Screen  and  Tactile  Interfaces  on  Performance  of  Virtual 
Unmanned  Aerial  Vehicle  Operation 
Protocol:  #L-680 

Principal  Investigator:  Dr.  Ming  Hou  (DRDC  Toronto) 

Research  Assistants:  Sheila  Young  and  Andrew  Sun 
Thrust  Code:  12PN03 


A.2  Glossary  of  Acronyms 


DRDC  Toronto 

Defence  Research  &  Development  Canada  -  Toronto 

DRDC  Valcartier 

Defence  R&D  Canada  -  Valcartier 

GCS 

Ground  Control  Station 

GUI 

Graphical  User  Interface 

EA-MAV 

Extremely  Agile  Micro  Aerial  Vehicle 

LCD 

Liquid  Crystal  Display 

MANCOVA 

Multivariable  Analysis  of  Covariance 

MAV 

Micro  Aerial  Vehicle 

NASA  TLX 

NASA  Task  Load  Index 

NPTC 

Non-playable  Target  Character 

PC 

Personal  Computer 

PDA 

Portable  Digital  Assistant 

SAGAT 

Situation  Awareness  Global  Assessment  Technique 

UMPC 

Ultra  Mobile  Personal  Computer 

UV 

Unmanned  Vehicle 

VGA 

Video  Graphics  Array 

VNCEP 

Virtual  Navigation  and  Collaboration  Experimentation  Platform 

A.3  Ethics  Committee  Proposal 

A.3.1  Background 

Micro  Aerial  Vehicles  (MAVs)  are  miniature  flying  machines  that  will  provide  “over-the-hill”  or 
“around-the-next-house”  reconnaissance  to  smaller  combat  elements  such  as  infantry  sections. 
The  device  that  will  become  the  Extremely  Agile  Micro  Aerial  Vehicle  (EA-MAV)  is  an  electric 
fixed  wing  aircraft  equipped  with  a  Video  Graphics  Array  (VGA)  video  sensor  and  an  audio 
sensor.  The  hardware  prototype  is  currently  under  development  at  Defence  Research  & 
Development  -  Valcartier  (DRDC  Valcartier).  Its  forward  flight  speed  can  vary  from  a  standstill 
to  a  running  pace.  In  hovering  mode  with  the  nose  pointed  vertically  up,  the  MAV  can  roll  about 
its  longitudinal  axis  to  scan  the  surroundings  with  the  video  sensor.  Using  onboard  electronics, 
the  MAV  will  automatically  stabilize  its  altitude.  While  the  MAV  is  recoverable,  depending  on 
the  tactical  situation,  the  soldier  may  discard  the  MAV  due  to  its  low  cost. 
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To  utilize  such  technology,  soldiers  will  require  a  Ground  Control  Station  (GCS)  to  interact  with 
whatever  flying  device  they  are  operating.  The  GCS  itself  is  subject  to  stringent  engineering 
criteria,  such  as  the  need  to  be  robust  enough  to  withstand  extreme  heat,  moisture  and 
acceleration.  The  interface  must  be  easy  to  learn  for  the  average  soldier,  and  be  intuitive  in 
function  so  that  time  is  not  lost  during  mission-critical  moments  in  deciphering  what  the  next 
control  sequence  should  be.  This  device  must  be  small  and  light  enough  to  not  seriously  hinder 
the  soldier  who  will  be  carrying  it.  To  date,  little  human  factors  work  has  been  done  on  what  is 
the  most  desired  method  for  controlling  such  a  device. 

A  number  of  hand  held  devices  have  been  developed  for  use  with  modem  Unmanned  Vehicles 
(UVs)  and  robots.  These  include  Personal  Digital  Assistants  (PDAs),  portable  laptops,  touch 
screen  tablets,  and  modular  systems.  Handhelds  may  either  serve  as  a  component  of  a  larger 
ground  control  station  as  the  operator  interface  or  video  interface,  or  they  can  function  as  the 
integrated  ground  control  station.  The  latest  trend  in  ground  UV  operation  has  seen  the  use  of 
game-style  hand  controllers  with  laptops  and  tablets.  Experiments  have  shown  that  game-style 
controllers  (Xbox  360)  have  led  to  faster  training  and  easier  operation  in  the  field  [1].  Within  this 
class  of  control  system  device,  the  type  of  input  technology  is  an  important  consideration.  The 
two  modem  input  methods  are  capacitive  or  resistive  touch-sensitive  display,  and  tactile  hardware 
buttons,  which  provide  haptic  feedback. 

To  control  the  EA-MAV  through  a  GCS,  the  soldier  requires  a  Graphical  User  Interface  (GUI)  to 
be  embedded  that  will  allow  them  to  interact  with  whatever  flying  device  they  are  operating.  The 
GUI  itself  is  subject  to  stringent  human  factors  engineering  criteria,  such  as  the  amount  of 
sensory  data  displayed,  screen  size,  resolution,  and  optimized  map  and  video  sensor  views. 
Additionally,  the  GUI  must  be  easy  to  leam  for  the  average  soldier,  and  be  intuitive  in  function 
and  display  so  that  time  is  not  lost  during  mission  critical  moments.  There  has  been  research  into 
over-layer  and  split  views  of  sensory  and  map  data  but  not  into  picture  in  picture  views. 

This  investigation  attempts  to  examine  the  differences  in  operator  performance  between  two 
interface  conditions.  The  first  condition  compares  touch-screen  and  tactile  input  methods,  while 
the  second  condition  looks  at  two  GUI  layouts:  separated  sensor  view  and  map  view,  versus 
combined  sensor  view  and  map  view.  The  findings  will  not  only  assist  in  choosing  the  optimal 
input  and  display  method  for  the  final  GCS  interface  system,  but  will  also  set  a  base  for  future 
experimentation  and  research  on  interfaces  using  this  platform. 

A.3.2  Purpose  of  Study 

This  study  aims  to  determine  how  the  performance  of  an  EA-MAV  operator  is  affected  by  two 
experimental  conditions:  (i)  Input  Control  Type  -  Touch  Screen  versus  Tactile  Button,  and  (ii) 
Display  Mode  (sensor  view  and  map  view)  -  Sequential  view  versus  Simultaneous  view. 

A.3.3  Selection  of  Human  Participants 

Approximately  36  civilian  and  military  volunteers  from  Defence  Research  &  Development  - 
Toronto  (DRDC  Toronto)  and  the  surrounding  community,  aged  18-60  with  self-reported  normal 
or  corrected-to-normal  vision,  including  normal  colour  vision,  will  be  recruited  by  email.  Both 
males  and  females  will  be  eligible  to  participate.  Participants  will  be  financially  compensated  for 
stress  allowance  according  to  DRDC  guidelines.  Investigators  will  not  serve  as  participants 
in  this  experiment. 
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A.3.4  Methodology 

Stimuli  and  Apparatus 

The  experiment  will  be  conducted  in  DRDC  Toronto,  Room  2000  with  normal  temperature.  The 
visual  stimuli  will  be  delivered  using  different  hardware  based  on  the  trial  session.  There  will  be 
six  separate  sessions,  all  conducted  on  one  day  in  succession.  Each  of  the  sessions  will  utilize  one 
of  the  different  input  methods  for  EA-MAV  control;  a  Viliv  S5  Ultra  Mobile  Personal  Computer 
(UMPC)  (see  Figure  A-l)  (a  product  by  Yukyung  Technologies  Coiporation  located  in  Anyang- 
si,  Gyeonggi-do  South  Korea),  and  a  standard  Personal  Computer  (PC)  with  a  keyboard  and 
mouse  that  will  be  used  as  a  baseline.  The  order  of  these  sessions  will  be  counterbalanced  across 
participants  to  account  for  learning  affects. 

The  first  session  will  be  conducted  on  a  Windows-based  workstation  with  a  Liquid  Crystal 
Display  (LCD)  with  screen  dimensions  41  cm  x  31  cm  and  a  resolution  1280  x  1024  pixels 
running  the  latest  version  of  the  Virtual  Navigation  and  Collaboration  Experimentation  Platform 
(VNCEP)  virtual  environment  software.  The  participant  will  be  seated  approximately  55  cm  away 
from  the  monitor.  In  this  session,  the  participant  will  control  the  playable  character  in  the  3- 
Dimensional  virtual  environment  using  the  keyboard  and  mouse.  The  mouse  will  control  the  gaze 
of  the  player  and  the  A,  W,  S  and  D  keys  will  control  the  movement  of  the  player.  The  participant 
will  use  the  same  workstation  to  view  the  two  GUIs  and  then  familiarize  themselves  with  the 
functionality  of  the  system  and  the  experiment  fly-task. 

In  the  second  session,  the  participant  will  use  the  same  workstation  as  above  to  complete  the  PC 
baseline  experiment  task. 


Figure  A-l :  Viliv  S 5  UMPC 
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In  all  the  following  sessions  (3-6)  the  participant  will  use  a  Viliv  S5  UMPC  to  control  the  EA- 
MAV  in  the  virtual  environment.  The  Viliv  S5  UMPC,  displayed  above  in  figure  A-l,  is  a 
commercially  available  ultra  mobile  personal  computer  with  a  screen  size  of  approximately 
10.5cm  x  6.3cm  and  a  screen  resolution  of  1024  x  600  pixels. 


Figure  A-2  -  The  Task  Mission 


Note,  in  Figure  A-2,  the  Way-marker  definitions  are  as  follows: 

0:  Starting  Location 

1,  2,  3:  Aligned  with  center  of  road  to  be  traveled  in  auto  mode 

A:  Access  -  Entry  Location.  Aligned  with  the  center  of  the  far  right  second  story 
window  to  fly  into  of  the  building 

E:  Exit  -  Finishing  Location.  Aligned  with  the  center  of  the  far  left  second  story 
window  to  fly  out  of  the  building 

Red  Dot:  Location  of  the  Non-playable  Target  Character  (NPTC) 

In  the  third  session,  the  screen  will  display  a  sensor  image  stream  from  the  virtual  EA-MAV’s 
VGA  sensor  and  on  a  separate  tab.  Once  selected,  the  screen  displays  a  bird’s-eye  view  of  the 
map  with  an  indicator  showing  the  relative  location  of  the  EA-MAV.  During  this  session  the 
participant  will  complete  the  task  mission  by  flying  through  the  way-markers  as  shown  in  figure 
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A-2,  and  will  control  the  EA-MAV  using  only  the  hardware  buttons  and  the  sequential  view  GUI. 
Once  complete,  the  user  will  be  given  the  task  questionnaire  and  the  situational  awareness 
questionnaire. 

In  the  fourth  session,  the  screen  will  display  a  sensor  image  stream  from  the  virtual  EA-MAV’s 
VGA  sensor  and  on  a  separate  tab.  Once  selected,  the  screen  will  display  a  bird’s-eye  view  of  the 
map  with  an  indicator  showing  the  relative  location  of  the  EA-MAV.  The  screen  will  display  the 
same  GUI  used  in  the  second  session.  During  this  session  the  participant  will  complete  the  task 
mission  by  flying  through  the  way-markers  as  shown  in  figure  A-2,  and  will  control  the  EA-MAV 
using  only  the  touch-sensitive  screen  and  none  of  the  tactile  buttons  and  the  sequential  view  GUI. 
Once  complete,  the  user  will  be  given  the  task  questionnaire. 

In  the  fifth  session,  the  screen  will  display  a  large  sensor  image  stream  from  the  virtual  EA- 
MAV’s  VGA  sensor  with  a  small  view  of  the  bird’s-eye  view  map  and  the  relative  location  of  the 
EA-MAV  and  on  a  separate  tab.  Once  selected,  the  screen  displays  a  large  bird’s-eye  view  of  the 
map  with  an  indicator  showing  the  relative  location  of  the  EA-MAV,  and  a  small  view  of  the 
virtual  EA-MAV’s  VGA  sensor.  During  this  session  the  participant  will  complete  the  task 
mission  by  flying  through  the  way-markers  as  shown  in  figure  A-2,  and  will  control  the  EA-MAV 
using  only  the  hardware  buttons  and  the  simultaneous  view  GUI.  Once  complete,  the  user  will  be 
given  the  task  questionnaire  (Annex  E)  and  the  situational  awareness  questionnaire  (Annex  F). 

In  the  sixth  session,  the  screen  will  display  a  large  sensor  image  stream  from  the  virtual  EA- 
MAV’s  VGA  sensor  with  a  small  view  of  the  bird’s-eye  view  map  and  the  relative  location  of  the 
EA-MAV  and  on  a  separate  tab.  Once  selected,  the  screen  displays  a  large  bird’s-eye  view  of  the 
map  with  an  indicator  showing  the  relative  location  of  the  EA-MAV,  and  a  small  view  of  the 
virtual  EA-MAV’s  VGA  sensor.  The  screen  will  display  the  same  GUI  used  in  the  fifth  session. 
During  this  session,  the  participant  will  complete  the  task  mission  by  flying  through  the  way- 
markers  as  shown  in  figure  A-2,  and  will  control  the  EA-MAV  using  only  the  touch-sensitive 
screen  and  none  of  the  tactile  buttons  and  the  sequential  view  GUI.  Once  complete,  the  user  will 
be  given  the  task  questionnaire  (Annex  E). 

The  participant  will  be  asked  to  navigate  the  EA-MAV  as  quickly  as  possible  from  the  launch  site 
following  the  defined  path  to  a  specified  building  in  the  scenario,  which  has  been  identified  as  a 
possible  threat.  Once  there,  the  participant  will  have  to  navigate  the  EA-MAV  accordingly  to 
enter  the  building  through  a  second  story  window.  The  participant  will  operate  the  EA-MAV  to 
view  the  NPTC  (illustrated  in  figure  A-3  below)  and  then  exit  through  the  window  on  the  far  left. 

During  the  second  baseline  session,  the  participant  will  follow  the  same  scenario  except  on  foot 
as  a  playable  character.  At  the  building  containing  the  possible  threat,  they  will  enter  through  a 
doorway  on  the  ground  level  directly  below  the  second  story  window  and  use  stairs  to 
reach  the  NPTC. 
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Figure  A-3:  Example  of  the  Sensor  Image  as  seen  by  the  EA-MA  V 

A.3.4.1  Experimental  Procedure 

The  experiment  will  look  at  the  two  types  of  control  interfaces;  touch  screen  and  tactile  as  the 
primary  independent  measure,  and  two  types  of  GUIs;  bird’s-eye  map  and  video  sensor  viewed  at 
the  same  time  on  a  single  window,  or  a  bird’s-eye  map  and  video  sensor  viewed  individually  in 
separate  windows.  The  dependent  measures  will  include  response  times  and  precision  to 
the  way-markers. 

There  will  be  six  sessions  completed  within  one  day.  The  first  session  (training)  will  last  about  60 
minutes.  The  following  five  sessions  will  last  about  15  minutes  each  with  5  minute  breaks  in- 
between,  followed  by  a  20  minute  question  and  debriefing  period.  The  total  time  is  set  to  be  3 
hours  with  breaks  included.  The  order  of  the  sessions  will  be  counter-balanced  to  account  for 
learning  affects. 

A.3.4.2  Data  Analysis 

The  dependent  variables  for  this  study  include  mission  completion  time,  the  EA-MAV  navigation 
path  precision,  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  [2]  on  situation 
awareness,  and  training  time.  They  are  described  in  section  3.2  of  reference  [3].  The  mission 
timings  will  be  compared  across  the  two  control  interfaces  and  the  2  display  views  to  determine 
maximum  performance  and  efficiency.  The  playable  character’s  and  the  EA-MAV’s  travel  path, 
which  is  along  the  defined  way-markers,  will  be  used  to  calculate  the  standard  deviation  and 
compared  across  the  two  interfaces,  the  two  graphical  interfaces  and  the  baseline.  The 
performance  will  be  measured  by  the  horizontal  distance  the  EA-MAV  travels  from  each  defined 
way-marker  and  the  number  of  times  the  participant  fails  a  mission.  The  mission  fail  rate  will 
also  be  measured  and  this  will  be  defined  by  the  number  of  times  the  participant  controlled 
playable  character  or  if  a  MAV  collides  with  a  stationary  object,  and/or  flies  off  the  map.  If  the 
participant  fails  a  mission,  they  may  retry  the  mission  by  selecting  the  ‘RESET’  button  and  they 
will  have  to  begin  the  mission  from  the  initial  starting  point. 

There  will  be  a  maximum  of  5  resets  within  the  limits  of  the  allotted  15-minute  task  time.  The 
number  of  clicks  (tactile  control)  and  taps  (touch  screen  control)  will  be  recorded  and  analyzed  to 
view  the  convenience  of  the  simultaneous  view  graphical  user  interface  versus  the  sequential 
view  graphical  user  interface.  Only  the  best  successful  mission  in  terms  of  time  and  performance 
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from  each  participant  will  be  used  when  data  are  analyzed.  The  participants  will  complete 
background  information  questionnaire  (Annex  D)  before  they  begin  training  and  a  post-task 
questionnaire  (Annex  E)  per  session.  The  participants  will  also  be  posed  with  a  situation 
awareness  questionnaire  (Annex  F)  the  first  time  they  complete  the  sequential  view  task  and  the 
first  time  they  complete  the  simultaneous  view  task.  The  background  information  will  be  used  to 
analyze  the  participants’  demographic  and  experience  with  touch  screen  and  tactile  button  input 
and  video  games.  The  post-task  questionnaire  will  contain  usability  and  workload  insensitivity 
questions  using  the  NASA  Task  Load  Index  (NASA  TLX)  [4]  10  scale  ratings.  The  data  from 
each  participant  will  be  entered  into  tables  2  and  3  shown  in  reference  [3]  and  used  to  analyze  the 
data  retrieved  by  each  individual,  by  each  group  and  all  the  groups  as  a  whole  subject. 

The  data  collected  will  be  analyzed  using  the  method  of  MANCOVA  (Multivariable  Analysis  of 
Covariance)  [5],  STATISTICA  [6]  from  StatSoft  will  be  used. 

The  significant  differences  between  the  means  will  be  tested.  Two  independent  variables  are 
control  type  (2  levels:  touch  screen  vs.  button)  and  display  mode  (2  levels:  sequential  vs. 
simultaneous).  Dependent  variables  are  completion  time,  route  error,  TLX  and  SAGAT.  The 
completion  time  in  PC  Baseline  mission  will  be  used  as  a  covariate.  A  two-by-two  within-group 
analysis  will  be  used  for  completion  time,  route  error  and  TLX.  Between-group  analysis  will  be 
used  for  SAGAT. 

The  interaction  effect  between  control  type  and  display  mode,  the  significance  level,  as  well  as 
the  correlation  of  the  dependent  variables  with  the  covariate  (completing  time  in  PC  Baseline 
mission)  will  be  provided  as  a  result  from  the  MANCOVA  [5]. 

A.3.4.3  Medical  Screening 

The  experiments  will  require  no  medical  screening.  Participants  will  be  asked  whether  they  have 
any  condition  that  would  preclude  working  with  a  computer  or  the  mobile  handheld  devices,  and 
whether  they  have  had  changes  in  caffeine  or  alcohol  intake.  Any  participants  with  such  a 
condition  will  not  participate. 

A.3.4.4  Physician  Coverage 

The  presence  of  a  physician  in  the  experiment  room  will  not  be  necessary.  All  sessions 
will  be  conducted  during  regular  working  hours. 

A.3.4.5  Roles  and  Qualifications  of  Team  Members 

Dr.  Ming  Hou,  Defence  Scientist  as  the  Investigator,  will  determine  all  procedures  for  the 
experiments  and  be  the  immediate  point  of  contact  for  all  trials.  The  research  assistants  (Sheila 
Y oung  and  Andrew  Sun)  will  contact  participants,  schedule  trials,  and  conduct  all  training  and 
experiment  sessions.  They  will  also  be  responsible  to  ensure  the  proper  set-up  and  use  of  the 
computers  and  equipment. 

A.3.4.6  Withholding  of  Information 

The  experiments  involve  minimum  withholding  of  information.  Participants  will  not  be  informed 
of  the  specific  hypotheses  and  stimulus  configurations  of  experiments.  They  will,  however, 
receive  thorough  and  truthful  explanations  of  the  purpose  and  procedures  of  the  experiments 
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before  beginning.  Participants  will  receive  a  debriefing  that  explains  the  specific  hypotheses 
examined  after  the  completion  of  the  experiment  by  all  participants.  The  participant  will  be 
informed  that  if  they  do  not  wish  their  data  to  be  used,  it  will  be  discarded. 

A.3.4.7  Risks  and  Benefits 

Risks 

The  experiments  will  be  performed  using  common  computer  equipment  in  an  office-like  setting, 
with  no  stressors  applied  to  participants.  A  possible  risk  to  the  subject  is  mild  fatigue  or  eyestrain 
from  performing  the  task  and/or  using  the  computer  and  devices.  To  minimize  this  effect, 
volunteer  participants  will  receive  a  break  between  training  and  test  sessions,  and  the 
experimental  task  will  be  made  as  engaging  as  possible.  The  nature  of  the  experiment,  including 
potential  risks,  will  be  explained  in  detail  to  participants  who  will  read  and  sign  a  form  to  indicate 
informed  consent.  The  experimental  data  concerning  each  subject  will  be  treated  as  confidential 
(‘Protected  B’  In  Accordance  With  Canadian  Forces  Security  Requirements)  and  will  not  be 
revealed  to  anyone  other  than  the  DRDC  Toronto  Investigator(s)  or  external  investigators  from 
the  sponsoring  agency  without  the  subject’s  consent,  except  as  data  unidentified  as  to  source. 

Benefits 

The  purpose  of  this  study  is  to  examine  the  difference  in  operator  performance  when  using  touch¬ 
screen  vs.  tactile  button  input  systems.  Specifically,  we  hope  to  learn  if  using  one  of  these  input 
methods  is  more  intuitive  and  effective  for  control  of  the  MAV  in  the  context  of  a  handheld 
portable  device.  These  differences  have  not  yet  been  examined  extensively  in  this  context  and 
therefore,  this  research  would  benefit  the  scientific  and  technological  communities  at  large  as 
well.  The  benefit  of  this  study  to  the  participant  is  to  learn  about  new  MAV  technologies  under 
development  for  use  in  reconnaissance. 

A.3.4.8  Potential  Conflicts  of  Interest 

All  funding  for  the  experiment  comes  from  Thrust  12PN03. 

A.3.4.9  Approximate  Time  Involvement 

The  experiment  will  consist  of  six  sessions  conducted  in  one  day,  with  a  60-minute  training 
session  and  with  each  of  the  remaining  5  sessions  being  approximately  1 5  minutes  in  duration 
each.  The  total  estimated  time  for  the  experiment  is  approximately  3  hours  including  the  breaks 
between  sessions. 

A.3.4.10  Participant  Remuneration 

Based  on  DRDC  Toronto’s  Experimental  Compensation  Calculator,  internal  subjects  will  be 
remunerated  a  total  of  $22.36  and  external  subjects  will  be  remunerated  a  total  of  $40.20  as  well 
as  travel  costs  to  an  amount  equivalent  to  the  cost  of  round-trip  TTC  fare  for  completion  of  this 
study  [7]. 

The  calculation  variables  are  as  follows: 

1)  Total  stress  level:  1  units 

2)  Total  number  of  hours  of  participation:  3  hours 

3)  Pay/stress  unit:  $22.36  for  internal  participants,  and  $40.20  as  well  as  travel  costs  to  an 
amount  equivalent  to  the  cost  of  round-trip  TTC  fare  for  external  participants. 
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Annex  B  Participant  Information  Sheet 


Comparison  between  Touch  Screen  and  Tactile  Interfaces  on  Performance  of  Virtual 
Unmanned  Aerial  Vehicle  Operation 

Background 

Extremely  Agile  Micro  Aerial  Vehicles  (EA-MAVs)  are  miniature  flying  machines  that  will 
provide  “over-the-hill”  or  “around-the-next-house”  reconnaissance  to  smaller  combat  elements 
such  as  infantry  sections.  It  is  essential  for  the  development  of  a  suitable  Ground  Control  System 
(GCS)  to  control  it.  Consequently,  there  is  a  requirement  for  human  factors  research  to  be  done 
on  the  subject  to  help  develop  and  deliver  the  most  efficient  user  interface.  One  significant  factor 
to  consider  in  designing  a  GCS  for  any  EA-MAVs  is  the  difference  between  the  touch  screen  and 
tactile  hardware  buttons  equipped  on  a  modem  type  of  handheld  input  device.  The  second  factor 
to  consider  is  having  map  and  sensor  views  combined  in  one  window  view  as  opposed  to  being 
split  into  separate  windows.  This  study  seeks  to  identify  the  differences  in  performance  and 
efficiency  of  the  operator  when  operating  an  EA-MAV  with  each  of  these  two  types  of  control 
interfaces  and  graphical  user  interfaces.  The  results  will  aid  in  the  design  of  the  final  EA-MAV 
GCS  interface  device. 

Task 

Y ou  will  be  asked  to  control  the  EA-MAV  using  a  device  and  navigate  it  through  urban  terrain. 
First,  you  will  navigate  on  a  defined  path  to  a  building  that  has  been  identified  as  a  possible 
threat.  Once  there,  you  will  have  to  enter  the  building.  It  is  given  that  the  character  in  the  room 
will  be  visible  on  the  screen.  The  path  to  travel  will  be  provided  to  you  on  a  reference  map.  The 
simulator  will  automatically  log  the  path  traveled  with  a  relative  efficiency  to  the  provided 
reference  path  and  the  number  of  commands  and  the  entire  tasks  timings;  these  are  then 
to  be  analyzed. 

At  the  beginning,  the  experiment  administrator  will  run  you  through  a  training  session,  which  will 
last  about  60  minutes.  You  will  operate  the  user  interface  via  Personal  Computer  (PC),  control  the 
device,  and  familiarize  yourself  with  the  mission  where  you  will  fly  through  a  short  flight  path 
through  the  terrain  to  familiarize  yourself  with  the  inputs  and  the  targets.  After  the  main  training 
session,  there  will  be  5-recorded  sessions.  Each  session  will  last  for  approximately  15  minutes. 
There  are  a  total  of  six  sessions,  all  conducted  in  one  day  with  short  breaks  in  between.  If  you 
have  any  questions  about  the  experiment,  the  experiment  administrator  will  answer  them  for  you. 
Should  you  feel  the  need  to  withdraw  from  the  experiment,  you  may  do  so  at  any  time. 

Rights  as  a  Participant 

Your  participation  in  this  study  is  completely  voluntary.  You  are  free  to  refuse  to  participate  and 
you  may  withdraw  your  consent  at  any  time,  in  which  case  your  participation  as  a  subject  will 
cease  immediately.  You  may  ask  questions  of  the  Investigator(s)  at  any  time  during  the 
training  session. 
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Confidentiality 

The  experimental  data  concerning  you  will  be  treated  as  confidential  (‘Protected  B’  in  accordance 
with  the  Canadian  Forces  Security  Requirements)  and  not  revealed  to  anyone  other  than  the 
Defence  Research  and  Development  Canada  -  Toronto  Investigator(s)  or  external  investigators 
from  the  sponsoring  agency  without  your  consent  except  as  data  unidentified  as  to  source. 

Benefits  of  Study 

The  findings  will  provide  a  starting  point  in  understanding  how  the  GCS  for  the  EA-MAV  can  be 
optimized.  The  differences  in  efficiency  and  performance  when  using  touch  screen  vs.  traditional 
tactile  controllers  have  not  been  examined  extensively  in  this  application.  Having  a  bird’s-eye 
map  and  a  camera  view  combined  in  one  window,  as  opposed  to  being  split  into  separate 
windows  and  viewed  individually,  have  also  not  been  examined  extensively  in  this  application. 

Risks 

For  each  of  the  sessions,  the  principal  risk  to  you  is  minor  eyestrain  and  fatigue  associated  with 
working  on  a  computer  and  using  handheld  devices. 

Compensation 

Internal  participants  will  receive  a  remuneration  of  $22.36,  and  external  participants  will  receive  a 
remuneration  of  $40.20  as  well  as  travel  costs  to  an  amount  equivalent  to  the  cost  of  round-trip 
TTC  fare  based  on  the  DRDC  Toronto’s  Experimental  Compensation  Calculator. 

Contact  Information 

Principal  Investigator: 

Dr.  Ming  Hou,  Phone:  416-635-2063,  Email:  Ming.Hou@drdc-rddc.gc.ca 


Chair,  DRDC  Human  Research  Ethics  Committee  (HREC): 

Dr.  JackP.  Landolt,  Phone:  416-635-2120,  Email:  Jack.Landolt@drdc-rddc.gc.ca 
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Annex  C  Consent  Form 


VOLUNTARY  CONSENT  FORM  FOR  HUMAN  SUBJECT  PARTICIPATION 


Protocol  Number:  #L-680 

Research  Project  Title:  Comparison  between  Touch  Screen  and  Tactile  Interfaces  on 
Performance  of  Virtual  Unmanned  Aerial  Vehicle  Operation 

Principal  Investigator:  Dr.  Ming  Hou  (DRDC  Toronto) 

Run  Director(s):  Sheila  Young  and  Andrew  Sun 

I, _ (name)  of _ (address  and 

phone  number)  hereby  volunteer  to  participate  as  a  subject  in  the  study  “Comparison  between 
Touch  Screen  and  Tactile  Interfaces  on  Performance  of  Virtual  Unmanned  Aerial  Vehicle 
Operation”  (Protocol  L-680).  I  have  read  the  information  letter  and  have  had  the  opportunity  to 
ask  questions  of  the  Investigator(s).  All  of  my  questions  concerning  this  study  have  been  fully 
answered  to  my  satisfaction.  However,  I  may  obtain  additional  information  about  the  research 
project  and  have  any  questions  about  this  study  answered  by  contacting  Dr.  Ming  Hou  at  416- 
635-2063  or  Dr.  Jack  Landolt  at  416-635-2000  Extension  2120. 

I  have  been  told  that  I  will  be  asked  to  participate  in  six  sessions  with  the  first  session  being 
approximately  60  minutes  in  duration  and  each  of  the  remaining  five  sessions  being 
approximately  15  minutes  in  duration  with  short  breaks  in-between  for  a  total  of  3  hours.  I  will 
fly  through  simulated  urban  terrain  using  different  input  methods  and  devices.  My  task  will  be  to 
navigate  the  vehicle  and  at  one  point  identify  a  target  character. 

I  have  been  told  that  the  experimental  data  concerning  me  will  be  treated  as  Protected  A  as 
appropriate,  and  not  revealed  to  anyone  other  than  the  DRDC-affiliated  Investigator(s)  or  external 
investigators  from  the  sponsoring  agency  without  my  consent  except  as  data  unidentified 
as  to  source. 

I  have  been  told  that  the  principal  risks  of  the  research  protocol  are  minor  eyestrain  and  fatigue 
associated  with  working  on  a  computer  and  using  handheld  devices.  Also,  I  acknowledge  that  my 
participation  in  this  study,  or  indeed  any  research,  may  involve  risks  that  are  currently  unforeseen 
by  DRDC  Toronto.  For  Canadian  Forces  (CF)  members  only:  I  understand  that  I  am  considered 
to  be  on  duty  for  disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in 
this  experiment.  This  duty  status  has  no  effect  on  my  right  to  withdraw  from  the  experiment  at 
any  time  I  wish  and  I  understand  that  no  action  will  be  taken  against  me  for  exercising  this  right. 
Furthermore,  I  understand  that  if  my  participation  in  this  study  results  in  a  medical  condition 
rendering  me  unfit  for  service,  I  may  be  released  from  the  CF. 
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I  understand  that  my  name  will  not  be  identified  or  attached  in  any  manner  to  any  publication 
arising  from  this  study.  Moreover,  I  understand  that  the  experimental  data  may  be  reviewed  by  an 
internal  or  external  audit  committee  with  the  understanding  that  any  summary  information 
resulting  from  such  a  review  will  not  identify  me  personally. 

I  understand  that  1  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without 
prejudice  or  hard  feelings  at  any  time.  Should  I  withdraw  my  consent,  my  participation  as  a 
subject  will  cease  immediately.  I  also  understand  that  the  Investigator(s),  or  their  designate 
responsible  for  the  research  project  may  terminate  my  participation  at  any  time,  regardless 
of  my  wishes. 

I  have  been  informed  that  the  research  findings  resulting  from  my  participation  in  this  research 
project  may  be  used  for  commercial  purposes. 

I  understand  that  as  a  CF  member  or  civilian  government  employee  participating  in  this  research 
project  during  work  hours,  I  am  entitled  to  a  remuneration  in  the  form  of  a  stress  allowance  for 
each  completed  session  for  a  total  amount  of  $22.36  if  I  complete  the  entire  research  project  as  set 
out  in  the  protocol.  I  understand  that  as  a  non-government  civilian  or  civilian  government 
employee  participating  in  this  research  project  during  non-work  hours,  I  am  entitled  to 
remuneration  that  incorporates  an  allowance  for  both  stress  and  my  commitment  of  time  for  each 
completed  session  for  a  total  amount  of  $40.20  as  well  as  travel  costs  to  an  amount  equivalent  to 
the  cost  of  round-trip  TTC  fare  if  I  complete  the  entire  research  project  as  set  out  in  the  protocol.  I 
also  understand  that  I  am  entitled  to  partial  remuneration  if  I  do  not  complete  all  of  the  sessions. 
Stress  remuneration  is  taxable.  My  Social  Insurance  Number  (SIN)  is  required  for  remuneration. 
T4A  slips  are  issued  only  for  amounts  combined  and  in  excess  of  $500.00  remuneration  per  year. 

I  have  informed  the  Principal  Investigator  that  I  am  currently  a  subject  in  the  following  other 

DRDC  research  project(s): _ (volunteer  to  cite  Protocol 

Number(s)  and  associated  Principal  Investigator(s)),  and  that  I  am  participating  as  a  subject  in  the 

following  research  project(s)  at  institutions  other  than  DRDC: _ 

(volunteers  to  cite  name(s)  of  institution(s)) 

Secondary  Use  of  Data:  I  understand  that  my  data  from  this  study  may  be  used  in  unidentified 
form  in  future  related  studies  provided  review  and  approval  have  been  given  by  DRDC  Human 
Research  Ethics  Committee. 

I  understand  that  by  signing  this  consent  form  I  have  not  waived  any  legal  rights  I  may  have  as  a 
result  of  any  harm  to  me  occasioned  by  my  participation  in  this  research  project  beyond  the  risks  I 
have  assumed.  Also,  I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may 
contact  any  of  the  individuals  mentioned  below  at  some  time  in  the  future  should  that  be  required. 

Volunteer’s  Name: _ . 

Signature: _ .Date: _ . 

Name  of  Witness  to  Signature: _ . 

Signature: _ .Date: _ . 
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Section  Head/Commanding  Officer’s  Signature  (see  Notes  below): 


.Date: 


Commanding  Officer’s  Unit: 


Contract  Manager’s  Signature  (see  Notes  below): 
_ .Datei _ 


Principal  Investigator: _ . 

Signature: _ .Date: _ . 

Notes: 

For  Canadian  Forces  (CF)  members  only:  I  understand  that  I  am  considered  to  be  on  duty  for 
disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in  this  study  and  I 
understand  that  in  the  unlikely  event  that  my  participation  in  this  study  results  in  a  medical 
condition  rendering  me  unfit  for  service,  I  may  be  released  from  the  CF  and  my  military  benefits 
apply.  This  duty  status  has  no  effect  on  my  right  to  withdraw  from  the  study  at  any  time  I  wish 
and  I  understand  that  no  action  will  be  taken  against  me  for  exercising  this  right. 

For  Military  personnel  on  permanent  strength  of  CFEME:  Approved  in  principle  by  Commanding 
Officer;  however,  members  must  still  obtain  their  Section  Head’s  signature  designating  approval 
to  participate  in  this  particular  research  project. 

For  other  military  personnel:  All  other  military  personnel  must  obtain  their  Commanding 
Officer’s  signature  designating  approval  to  participate  in  this  research  project. 

For  civilian  employees  at  DRDC:  Signature  of  Section  Head  of  appropriate  research  centre  is 

required  designating  that  volunteer  subject  is  considered  either  to  be  at  work  ( _ (initials 

of  volunteer))  or  participating  on  their  own  time  ( _ (initials  of  volunteer))  and  that 

approval  has  been  given  to  participate  in  this  research  project. 

For  civilian  contractors  working  at  a  DRDC  Centre:  Signature  of  the  contract  manager  must  be 
obtained  indicating  they  are  aware  of  the  volunteer’s  intent  to  participate  in  this  research  project. 

FOR  SUBJECT  ENQUIRY  IF  REQUIRED: 

Should  I  have  any  questions  or  concerns  regarding  this  project  before,  during  or  after 
participation,  I  understand  that  I  am  encouraged  to  contact  the  appropriate  DRDC  research  centre 
cited  below.  This  contact  can  be  made  by  surface  mail  at  this  address  or  by  phone  or  email  to  any 
of  the  DRDC  numbers  and  addresses  of  individuals  listed  below: 
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Defence  R&D  Canada-Toronto 

1133  Sheppard  Avenue  West 
PO  Box  2000 

Toronto,  Ontario,  M3M  3B9 

Principal  Investigator  or  Principal  DRDC  Investigator: 

Dr.  Ming  Hou,  DRDC  Toronto,  416-635-2063,  Ming.Hou@drdc-rddc.gc.ca 
Chair,  DRDC  Human  Research  Ethics  Committee  (HREC): 

Dr.  Jack  P.  Landolt,  DRDC  Toronto,  416-635-2120,  Jack.Landolt@drdc-rddc.gc.ca 
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Annex  D  Participant  Background  Questionnaire 


Participant  #: _  Group  #: _  Administrator: _  Trainer: 

The  following  information  will  only  be  retained  for  the  purposes  of  this  study,  and  will  not  be 
disclosed  to  any  other  individual  or  organization. 

Please  circle  the  appropriate  responses: 

1-  Age: _ 

2.  Gender:  Male  or  Female 

3.  Have  you  ever  had  experience  in  Unmanned  Vehicle  (e.g.,  Call  of  Duty,  Remote 
Control  toys,  etc.)  control?  Yes  or  No 

If  Yes,  please  indicate  approximately  the  number  of  hours  of  experience: 

Less  than  29  hours  30-49  hours  50-99  hours  More  than  1 00  hours 

4.  Have  you  ever  played  video  games?  Yes  or  No 

If  Yes,  please  indicate  approximately  the  number  of  hours  played  in  total: 

<29  hours  30-49  hours  50-99  hours  >100  hours 

Please  list  console(s): _ 

5.  Have  you  ever  had  experience  with  Touch  Screen  interfaces?  Yes  or  No 
If  Yes,  please  indicate  approximately  the  number  of  hours  of  experience: 

<29  hours  30-49  hours  50-99  hours  >100  hours 

6.  Have  you  ever  had  experience  with  Button  Control  interfaces?  Yes  or  No 
If  Yes,  please  indicate  approximately  the  number  of  hours  of  experience: 

<29  hours  30-49  hours  50-99  hours  >100  hours 

7.  Have  you  ever  had  experience  with  handheld  devices?  Yes  or  No 

If  Yes,  please  indicate  approximately  the  number  of  hours  of  experience: 

<29  hours  30-49  hours  50-99  hours  >100  hours 

Please  list  device(s): _ 

- Training  Admin  Section - 


G 1 :  PC 

;  To-Si 

;  Ta-Si 

;  To-Se 

;  Ta-Se  : 

G2:  PC 

;  Ta-Si 

;  To-Se 

;  Ta-Se 

;  To-Si  : 

G3:  PC 

;  To-Se 

;  Ta-Se 

;  To-Si 

;  Ta-Si  : 

G4:  PC 

;  Ta-Se 

;  To-Si 

;  Ta-Si 

;  To-Se  : 

DRDC  Toronto  TR  2010-075 


49 


Annex  E  Subjective  Measure  Rating  Form 


Workload  Measure 

As  noted  above  in  the  protocol,  a  form  will  be  used  to  obtain  objective  measures  of 
workload.  Subjective  ratings  in  the  form  of  a  modified  NASA  TLX  [3]  10  scale  will  be  used. 
These  rating  forms  will  be  tuned  for  each  participant  and  for  each  of  the  participant’s  critical 
tasks.  Note  -because  of  the  relatively  small  sample  size  in  this  experiment,  analyses  of  cross-  and 
inter-correlations  are  not  justified  and  paired  comparisons  will  not  be  performed. 

Instructions:  Read  and  understand  the  Rating  Scale  Definitions  table  below.  Then  use 
the  rating  scale  on  page  20  to  mark  your  results. 


RATING  SCALE  DEFINITIONS 

Title 

Endpoints 

Descriptions 

MENTAL  DEMAND 

Low/High 

How  much  mental  and  perceptual  activity  was  required 
(e.g.,  thinking,  deciding,  calculating,  remembering, 
looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

PHYSICAL 

DEMAND 

Low/High 

How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the 
task  easy  or  demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low/High 

How  much  time  pressure  did  you  feel  due  to  the  rate  or 
pace  at  which  the  tasks  or  task  elements  occurred?  Was 
the  pace  slow  and  leisurely  or  rapid  and  frantic? 

EFFORT 

Low/High 

How  hard  did  you  have  to  work  (mentally  and 
physically)  to  accomplish  your  level  of  performance? 

PERFORMANCE 

Good/Poor 

How  successful  do  you  think  you  were  in  accomplishing 
the  goals  of  the  task  set  by  the  experiment  administrator 
(or  yourself)?  How  satisfied  were  you  with  your 
performance  in  accomplishing  these  goals? 

FRUSTRATION 

LEVEL 

Low/High 

How  insecure,  discouraged,  irritated,  stressed,  and 
annoyed  versus  secure,  gratified,  content,  relaxed  and 
complacent  did  you  feel  during  the  task? 
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Instructions:  Using  the  rating  scales  below  and  the  definitions  provided,  mark  each  scale  to 
indicate  the  degree  of  workload  you  experienced  while  completing  the  indicated  task.  Take 
note  that  with  the  exclusion  of  the  performance  scale  the  ratings  are  low  to  high,  where  as  the 
performance  scale  is  good  to  poor. 

Task:  Establish  the  EA-MAV  in  a  position  inside  of  the  target  building  to  allow  the  operator 
to  view  a  non-playable  target  character. 


MENTAL  DEMAND 


Low  1  3  J 

PHYSICAL  DEMAND 

1  1  1  1  1  1  1  1  1  1 

5 

1  1 

7 

lilil 

9  High 

lilil 

Low  1  3  ! 

5 

7 

9  High 

TEMPORAL  DEMAND 

1  1  1  1  1  1  1  1  1  1 

1  1 

lilil 

hill 

Low  1  3  ! 

5 

7 

9  High 

EFFORT 

1  1  1  1  1  1  1  1  1  1 

I  | 

hill 

lilil 

Low  1 3 l 

PERFORMANCE 

1  1  1  1  1  1  1  1  1  1 

5 

1  1 

7 

lilil 

9  High 

hill 

Good  1  3  ! 

5 

7 

9  Poor 

FRUSTRATION 

1  1  1  1  1  1  1  1  1  1 

1  1 

lilil 

hill 

Low  1  3  ! 

5 

7 

9  High 
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Annex  F  Situational  Awareness  Measure  Questionnaire 


As  noted  above  in  the  protocol,  a  questionnaire  will  be  used  to  obtain  objective  measures  of 
situational  awareness.  Two  questions  regarding  the  relative  location  and  the  image  viewed  from 
the  video  sensor  at  that  moment  would  be  asked. 


1.  Which  image  closely  represents  the  current  location  of  the  EA-MAV  on  the  map,  i.e.  the 
location  on  the  map,  at  the  time  the  mission  is  paused?  Please  only  circle  one  image’s 
corresponding  letter. 


a.  c. 
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2.  Which  image  closely  represents  the  current  image  from  the  sensor  of  the  EA-MAV,  i.e. 
the  sensor  view,  at  the  time  the  mission  is  paused?  Please  only  circle  one  image’s 
corresponding  letter. 
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Acronyms  and  initialisms 


ANOVA 

Analysis  of  Variance 

CF 

Canadian  Forces 

DRDC 

Defence  Research  &  Development  Canada 

EA-MAV 

Extremely  Agile  Micro  Aerial  Vehicle 

GCS 

Ground  Control  Station 

GUI 

Graphical  User  Interface 

MAV 

Micro  Aerial  Vehicle 

NASA 

National  Aeronautics  and  Space  Administration 

NPTC 

Non-playable  Target  Character 

PC 

Personal  Computer 

PDA 

Personal  Digital  Assistant 

R&D 

Research  &  Development 

SAGAT 

Situational  Awareness  Global  Assessment  Technique 

TLX 

Task  Load  Index 

UAV 

Unmanned  Aerial  Vehicle 

UMPC 

Ultra  Mobile  Personal  Computer 

UV 

Unmanned  Vehicle 

VNCEP 

Virtual  Navigation  and  Collaboration  Experimentation  Platform 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 
Management 

R&D 

Research  &  Development 
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experiment  and  navigated  a  virtual  EA-MAV  through  specified  way-points  in  an  urban 
area  and  a  building.  Subjects’  performance  was  measured  against  six  dependent 
variables:  (1)  situation  awareness,  (2)  display  switch  frequency  (between  sensor  view  and 
map  view),  (3)  task  completion  time,  (4)  mental  workload,  (5)  trajectory  error  of  the  flight, 
and  (6)  training  time.  The  results  revealed  that  the  simultaneous  display  and  the  touch 
screen  control  are  the  optimal  design  methods  for  the  handheld  interface  to  be  used  easily 
when  maintaining  situation  awareness.  The  findings  provided  guidance  for  designing 
operator  interfaces  on  handheld  devices  and  further  facilitated  the  development  of  a 
statement  of  requirements  of  EA-MAV  systems 

(U)  L’emploi  des  Extremement  agile  Micro  Aerial  Vehicules  (EA-MAVs)  entraine  la  necessite 
de  developper  un  instrument  de  controle  petit  et  leger  qui  n’empecherait  pas  les  soldats 
qui  I’utilise.  Cet  exigence  porte  un  probleme  avec  la  conception  d’un  interface  operateur 
efficace  pour  I’integration  avec  I’instrument  de  controle.  II  n’y  a  pas  beaucoup  de 
recherche  dans  le  domaine  des  facteurs  humains  qui  mettre  au  point  le  methode  le  plus 
efficace  pour  controler  un  EA-MAV  en  utilisant  un  appareil  portatif.  Pour  examiner  les 
methodes  de  conception  pour  le  developpement  d’un  interface  avec  un  fonctionnement 
intuitif  et  qui  est  facile  pour  la  moyenne  des  soldats  a  apprendre,  RDDC  Toronto  a  mene 
une  experience  pour  evaluer  de  modes  d’affichage  et  de  methodes  de  controle  d’entree. 
Le  mode  d’affichage  a  compare  un  affichage  combine  en  vue  de  capteurs  et  vue  sur  la 
carte  (simultanee),  avec  un  ecran  qui  separe  les  deux  points  de  vue  (sequentiel).  Le 
methode  de  controle  d’entree  a  compare  un  ecran  tactile  avec  des  boutons  tactiles. 
Quarante  quatre  (44)  sujets  ont  participe  a  I'experience,  ou  ils  ont  navigue  un  EA-MAV 
virtuel  par  un  chemin  specifie  dans  un  espace  urbain  et  un  batiment.  Les  six  variables 
dependantes  mesurees  de  la  performance  des  sujets  ont  ete  (1)  conscience  de  la 
situation  (2)  la  frequence  de  commutation  d’affichage  (entre  la  vue  du  capteur  et  la  carte), 
(3)  le  temps  d'execution  des  taches,  (4)  charge  de  travail  mental,  (5)  erreur  de  trajectoire 
du  vol,  et  (6)  le  temps  de  formation.  Les  resultats  ont  revele  que  I'affichage  simultane  et 
I'ecran  tactile  de  controle  sont  les  conditions  les  plus  avantageuses  en  concernant  la 
facilite  d’usage  en  conservant  conscience  de  la  situation  pour  I’interface  portatif.  Les 
conclusions  a  fourni  des  orientations  pour  la  conception  d’un  interface  operateur  a  utiliser 
en  conservant  conscience  de  la  situation.  Ces  resultats  fournissent  un  guide  pour  la 
conception  d’interface  d’operateur  dans  dispositifs  portables  and  a  facilite  le 
developpement  des  conditions  des  systemes  d’  EA-MAV. 
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(U)  micro  aerial  vehicle;  unmanned  vehicle;  remotely  operated  vehicle;  operator  interface; 
interface  design;  handheld  device;  mobile  computer;  display  format;  simultaneous  display; 
sequential  display;  command  input  method;  tactile  button,  touch  screen;  measure  of 
effectiveness;  measure  of  performance;  situation  awareness;  workload 
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